About this research groupWe are active in international health systems research and in particular in health benefit package design. Our activities have led to the development of ‘evidence-informed deliberative processes’ (EDPs) which we now implement in close collaboration with Ministries of Health in a number of countries, including Ghana, Iran, Kazachstan, Moldova and Pakistan. read more
About this research group
We are active in international health systems research and in particular in health benefit package design.
Our activities have led to the development of ‘evidence-informed deliberative processes’ (EDPs) which we now implement in close collaboration with Ministries of Health in a number of countries, including Ghana, Iran, Kazachstan, Moldova and Pakistan.
Apart from our work on benefit package design we undertake studies to inform health policy decisions in HIV/AIDS programming, maternal health, and scaling-up of surgery.
We support decision-makers in countries around the world in the development and implementation of legitimate processes for health policy making, in particular for health benefit package design.
Health benefit package design is an intrinsically complex and value-laden process, in which multiple criteria beyond cost-effectiveness are important, and stakeholders often justifiably disagree about the relative importance of these criteria. Yet, present theory and methods for priority setting do not reflect this, and are typically technocratic in nature. This risks to undermine the legitimacy of decisions.
We consider legitimacy, or perceived reasonableness of decisions, as the key challenge of priority setting in the coming decades. HTA-related organizations need to manoeuver between the rapid development of new technologies and ever-expanding pressures on health budgets on the one hand, and strong stakeholder voices demanding access to these technologies on the other hand. HTA organisations are increasingly scrutinized on how they make decisions, including their processes and the evidence used.
We have developed evidence-informed deliberative processes (EDPs) for HTA-related organisations to improve the legitimacy of their benefit package design.
Guidance for HTA bodiesThe use of evidence-informed deliberative processes.
This is the second version of the guide. It provides support to HTA bodies to implement EDPs on the basis of 100 practical questions.
EDPs: a practical guide for HTA bodies
This is the second version of the guide. It provides support to HTA bodies to implement EDPs on the basis of 100 practical questions.
EDPs are presently employed by national health authorities in Ghana, Iran, Kazakhstan, Moldova, Pakistan and Ukraine to design or revise their health their benefit packages. Its principles were used to design the health benefit package in Thailand, the Netherlands, and West Java province in Indonesia. EDPs are recommended in the guidance on HTA of the World Health Organization.
The guide can be downloaded here.
The EDP framework has a number of particular features, which it distinguishes from other HTA frameworks:
- EDPs consider HTA as a political and intrinsically complex undertaking, and has a strong emphasis on deliberative processes involving relevant stakeholders. We have developed approaches that stimulate debate and deepen the argumentation. This is different from many other HTA frameworks which are often limited to evidence collection and compilation.
- The EDP framework combines different theories and best HTA practices on HTA – it is not proposing anything new. In terms of theory, EDPs are based on rational decision-making (as in multi-criteria decision analyses - MCDA) and fair decision-making (as in accountability for reasonableness – A4R). In terms of practice, it is based on a collection of best practices of HTA bodies around the world.
- The framework provides a practical stepwise approach that covers the whole HTA process, i.e. situational analysis, installation of an appraisal committee, selecting health technologies and criteria, assessment, appraisal, and communication and appeal (see Figure below). Other HTA frameworks typically concentrate on selected steps only, e.g. evidence collection.
- EDPs take the current decision-making context as the starting point, and offers specific advice depending on the level of HTA development. The EDP framework is also relevant for countries that have not (yet) established an HTA body.
- EDPs can be considered as a practical application of the WHO ‘Fair choices’ framework by Norheim et al., in terms of the stepwise approach to achieve UHC.
- WHO EMRO recommends the use of EDPs for HTA bodies. It has been published and commented on extensively.
See the six steps of Evidence Informed Deliberative Processes in the Figure below.
We provide on-going support to the Ministry of Health on the institutionalisation of HTA in the country.
EDPs in Ghana
In Ghana, Gavin Surgey provides on-going support (through Access and Delivery Partnership/PATH) to the Ministry of Health on the institutionalisation of HTA in the country. Recent achievements include supporting the establishment of the HTA advisory committees.
Current work includes:
- The development of the processes, and capacity building, for Health Technology Assessment (HTA) using Evidence-Informed Deliberative Processes (EDPs) as the guiding framework.
- Working with the HTA committees on the assessment of the performance of selected interventions reimbursed by the National Health Insurance Agency (NHIA).
- A review of the historical and current legislation, regulation and policy on HTA and priority setting in Ghana.
We have implemented EDPs to improve priority setting of HIV/AIDS health technologies at the province level.
EDPs in Indonesia
We have implemented EDPs to improve priority setting of HIV/AIDS health technologies at the province level and are now supporting the implementation of EDPs at the district level in Indonesia.
- Tromp N, Prawiranegara R, Siregar A, Wisaksana R, Pinxten L, Pinxten J, Lesmana Putra A, Kurnia Sunjaya D, Jansen M, Hontelez J, Maurits S, Maharani F, Bijlmakers L, Baltussen R. Translating international HIV treatment guidelines into local priorities in Indonesia. Trop Med Int Health. 2018 Mar;23(3):279-294
We are currently supporting the implementation of EDPs in Iran, targeting 30 conditions across 6 disease areas.
EDPs in Iran
We are currently supporting the implementation of EDPs in Iran, in close collaboration with the High Council for Health Insurance (HCHI) Ministry of Health, to define their Health Insurane Benefit Package at the national level, targeting 30 conditions across 6 disease areas.
The activities include:
- Support on the implementation of evidence-informed deliberative processes in the country, i.e. remote supervision of a team of five analysts. This involves bi-weekly conference calls, webinars for capacity building, and country visits where possible / needed.
- Identify stakeholders and develop mechanism to involve stakeholders in the decision-making process;
- Advise HCHI on how to prepare evidence for deliberation;
- Provide guidance on considering additional criteria in health benefit package design in Iran
- Prepare workshops at HCHI level involving > 100 stakeholders on benefit package design;
- Development of recommendations on the in- or exclusion of services in 20 conditions across six health domains for the HCHI.
In Kazakhstan we are working in a World bank project on the development of HTA.
EDPs in Kazakhstan
In Kazakhstan we are working in a World bank project with the Ministry of Health on the development of HTA. The aim is developing a priority-setting tool based on evidence-informed deliberative processes (EDPs) to determine the State Guaranteed Benefit Package (SGBP) and the package of the Mandatory Social Health Insurance (MSHI), using HTA
This involves the use of all EDPs steps: undertaking a situational analysis; formation of an advisory committee; definition of a relevant set of criteria for priority setting; assessment of performance of interventions; arrangement of a deliberative process on priorities; implementation plan of rationing decisions and M&E framework.
Specifically, we undertake an assessment of 25 priority health technologies and develop a methodology for its appraisal using qualitative and quantitative MCDA approaches.
We support the government of Moldova to plan and implement the most suitable system for Health Technology Assessment, using EDPs.
EDPs in Moldova
The World Bank contracted Radboudumc to support the government of Moldova to plan and implement the most suitable system for Health Technology Assessment (HTA), using Evidence-Informed Deliberative Processes (EDPs) as the guiding framework.
This study falls under the Toward Universal Health Coverage Project which is a joint project implemented by the World Bank and financed by the Swiss Agency for Development and Cooperation (SDC). Our study runs between November 2019 and December 2021 and has three main tasks: conducting a needs assessment in terms of stakeholders to involve in HTA development, identifying HTA capacity and skills, mapping HTA systems in European countries to draw lessons for Moldova, and then developing an implementation plan, including a roadmap for short- and long-term implementation.
In Pakistan we support the implementation process of the DCP3 project using an evidence-informed deliberative process.
EDPs in Pakistan
In Pakistan we support the implementation process of the Disease Control Priorities 3 (DCP3) project using an evidence-informed deliberative process. DCP3 responds to the increasing need of low- and lower middle-income countries for technical guidance and support in benefit package design and in accelerating progress towards UHC.
The EDP process was embedded in a broader institutional effort around health benefit packages in Pakistan, initiated by a joint WHO-EMRO and DCP3 secretariat mission visit to Pakistan in Jan 2019. The six steps of evidence-informed deliberative processes (EDPs) were operationalized for UHC benefit package design in Pakistan, providing EDP instructions and templates for the prioritization of health services with regard to the installation of an advisory committee; identification of decision criteria; selection of health services for scoping and assessment; assessment of health services; and appraisal of health services.
We provide on-going support to the Ministry of Health on the institutionalisation of HTA in the country and capacity building.
EDPs in Tanzania
In Tanzania, Gavin Surgey provides on-going support (through ADP/PATH) to the Ministry of Health on the institutionalisation of HTA in the country and capacity building.
Recent achievements include supporting the establishment of the HTA advisory committee, the development of their HTA process guidance using Evidence-Informed Deliberative Processes (EDPs) as the guiding framework and, an analysis of selected medicines reimbursed by the National Health Insurance Service.
- Bijlmakers L, Jansen M, Boer B, van Dijk W, Groenewoud S, Zwaap J, Helderman JK, van Exel J, Baltussen R. Increasing the Legitimacy of Tough Choices in Healthcare Reimbursement: Approach and Results of a Citizen Forum in The Netherlands. Value Health. 2020 Jan;23(1):32-38. doi: 10.1016/j.jval.2019.07.015. Epub 2019 Sep 20.
- Jansen M, Baltussen R, Bijlmakers L, Tummers M. The Dutch Citizen Forum on public reimbursement of health care: a qualitative analysis of opinion change. Int J Health Pol Manag 2020;x(x),1-10. Doi 10.34172/ijhpm.2020.81
- Kapiriri L, Baltussen R, Oortwijn W. Implementing evidence-informed deliberative processes in health technology assessment: a low income country perspective. Int J Technol Assess Health Care. 2020 Jan 16:1-5. doi: 10.1017/S0266462319003398. [Epub ahead of print]
- Oortwijn W, Jansen M, Baltussen R. Use of Evidence-Informed Deliberative Processes by Health Technology Assessment Agencies Around the Globe. Int J Health Policy Manag. 2020 Jan 1;9(1):27-33. doi: 10.15171/ijhpm.2019.72.
- Oortwijn W, Van Oosterhout S, Kapiriri L. Application of evidence-informed deliberative processes in health technology assessment in low- and middle-income countries. International Journal of Technology Assessment in Health Care, 2020; 1-5. doi:10.1017/S0266462320000549
- Baltussen R, Marsh K, Thokala P, Diaby V, Castro H, Cleemput I, Garau M, Iskrov G, Olyaeemanesh A, Mirelman A, Mobinizadeh M, Morton A, Tringali M, van Til J, Valentim J, Wagner M, Youngkong S, Zah V, Toll A, Jansen M, Bijlmakers L, Oortwijn W, Broekhuizen H. Multicriteria Decision Analysis to Support Health Technology Assessment Agencies: Benefits, Limitations, and the Way Forward. Value Health. 2019 Nov;22(11):1283-1288. doi: 10.1016/j.jval.2019.06.014. Epub 2019 Oct 16.
- de Bresser I, Remers TEP, Wieland MWM, Prawiranegara R, Siregar AYM, Baltussen R. Prioritizing HIV/AIDS prevention strategies in Bandung, Indonesia: A cost analysis of three different HIV/AIDS interventions. PLoS One. 2019 Aug 15;14(8):e0221078. doi: 10.1371/journal.pone.0221078. eCollection 2019.
- Chen Y, Chi X, He Y, Wei Y, Oortwijn W, Shi L. Mapping of Health Technology Assessment in China: Situation Analysis and International Comparison. Int J Technol Assess Health Care. 2019;35(5):401-407. doi: 10.1017/S0266462319000709. Epub 2019 Oct 28.
- Jansen MPM, Bijlmakers L, Baltussen R, Rouwette EA, Broekhuizen H. A sustainable approach to universal health coverage. Lancet Glob Health. 2019 Aug;7(8):e1013. doi: 10.1016/S2214-109X(19)30252-9. No abstract available.
- Oortwijn W, Klein P.Addressing Health System Values in Health Technology Assessment: The Use of Evidence-Informed Deliberative Processes. Int J Technol Assess Health Care. 2019 Apr 8:1-3. doi: 10.1017/S0266462319000187. [Epub ahead of print]
- Pichler F, Oortwijn W, Ruether A, Trowman R. Defining capacity building in the context of HTA: a proposal by the HTAi Scientific Development and Capacity Building Committee. Int J Technol Assess Health Care. 2019;35(5):362-366. doi: 10.1017/S0266462319000631. Epub 2019 Sep 11.
- Pichon-Riviere A, Garcia-Marti S, Oortwijn W, Augustovski F, Sampietro-Colom L. Defining the Value of Health Technologies in Latin America: Developments in Value Frameworks to Inform the Allocation of Healthcare Resources. Int J Technol Assess Health Care. 2019 Jan;35(1):64-68. doi: 10.1017/S0266462319000072.
- Reckers-Droog V, Jansen M, Bijlmakers L, Baltussen R, Brouwer W, van Exel J. How does participating in a deliberative citizens panel on healthcare priority setting influence the views of participants? Health Policy. 2019 Dec 6. pii: S0168-8510(19)30280-5. doi: 10.1016/j.healthpol.2019.11.011. [Epub ahead of print]
- Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, Baltussen R. The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective. BMJ Glob Health. 2019 Jan 25;4.
- Baltussen R, Jansen M, Bijlmakers L. Stakeholder participation on the path to universal health coverage: the use of evidence-informed deliberative processes. Trop Med Int Health. 2018 Oct;23(10):1071-1074. doi: 10.1111/tmi.13138.
- Jansen MPM, Baltussen R, Bærøe K. Stakeholder Participation for Legitimate Priority Setting: A Checklist. Int J Health Policy Manag. 2018 Jul 4;7(11):973-976.
- Oortwijn W, Sampietro-Colom L, Habens F, Trowman R. How can health systems prepare for new and emerging health technologies? The role of horizon scanning revisited. Int J Technol Assess Health Care. 2018 Jan;34(3):254-259. doi: 10.1017/S0266462318000363. Epub 2018 Jun 11.
- Tromp N, Prawiranegara R, Siregar A, Wisaksana R, Pinxten L, Pinxten J, Lesmana Putra A, Kurnia Sunjaya D, Jansen M, Hontelez J, Maurits S, Maharani F, Bijlmakers L, Baltussen R. Translating international HIV treatment guidelines into local priorities in Indonesia. Trop Med Int Health. 2018 Mar;23(3):279-294.
- Baltussen R, Jansen MPM, Bijlmakers L, Grutters J, Kluytmans A, Reuzel RP, Tummers M, der Wilt GJV. Value Assessment Frameworks for HTA Agencies: The Organization of Evidence-Informed Deliberative Processes. Value Health. 2017 Feb;20(2):256-260.
- Baltussen R, Jansen MP, Bijlmakers L, Tromp N, Yamin AE, Norheim OF. Progressive realisation of universal health coverage: what are the required processes and evidence? BMJ Glob Health. 2017 Aug 22;2(3).
- Baltussen R, Mitton C, Danis M, Williams I, Gold M. Global Developments in Priority Setting in Health. Int J Health Policy Manag. 2017 Jan 28;6(3):127-128. doi: 10.15171/ijhpm.2017.10.
- Goetghebeur M, Castro-Jaramillo H, Baltussen R, Daniels N. The art of priority setting. Lancet. 2017 Jun 17;389(10087):2368-2369.
- Jansen MP, Baltussen R, Mikkelsen E, Tromp N, Hontelez J, Bijlmakers L, van der Wilt GJ. Evidence-Informed Deliberative Processes - Early Dialogue, Broad Focus and Relevance: A Response to Recent Commentaries. Int J Health Policy Manag. 2017 Jul 22;7(1):96-97. doi: 10.15171/ijhpm.2017.88.
- Mikkelsen E, Hontelez JA, Jansen MP, Bärnighausen T, Hauck K, Johansson KA, Meyer-Rath G, Over M, de Vlas SJ, van der Wilt GJ, Tromp N, Bijlmakers L, Baltussen RM. Evidence for scaling up HIV treatment in sub-Saharan Africa: A call for incorporating health system constraints. PLoS Med. 2017 Feb 21;14(2).
Sorted on year (2009-2015) and first author.
- M Jansen, N Tromp, R Baltussen PrEP: why we are waiting. Lancet HIV. 2015.
- Norheim, O.F., Baltussen, R.M., Johri, M., Chisholm, D., Nord, E., Brock, D., Carlsson, P., Cookson, R., Daniels, N., Danis, M., Fleurbaey, M., Johansson, K.A., Kapiriri, L., Littlejohns, P., Mbeeli, T., Rao, K.D., Edejer, T.T. & Wikler, D. Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Eff Resour Alloc. 2014 Aug 29;12:18. doi: 10.1186/1478-7547-12-18. eCollection 2014.
- Tromp N, Prawiranegara R, Subhan Riparev H, Siregar A, Sunjaya D, Baltussen R Priority setting in HIV/AIDS control in West Java Indonesia: an evaluation based on the accountability for reasonableness framework. Health Policy Plan. 2014 Apr 15.
- Venhorst K, Zelle SG, Tromp N, Lauer JA. Multi-criteria decision analysis of breast cancer control in low- and middle- income countries: development of a rating tool for policy makers. Cost Eff Resour Alloc. 2014 May 17;12:13. doi: 10.1186/1478-7547-12-13. eCollection 2014.
- Tanios N, Wagner M, Tony M, Baltussen R, van Til J, Rindress D, Kind P, Goetghebeur MM; International Task Force on Decision Criteria. Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care. 2013 Oct;29(4):456-65. doi: 10.1017/S0266462313000573.
- Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, Goetghebeur MM. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012 Jul 18;10(1):9.
- Mirelman A, Mentzakis E, Kinter E, Paolucci F, Fordham R, Ozawa S, Ferraz M, Baltussen R, Niessen LW. Decision-Making Criteria among National Policymakers in Five Countries: A Discrete Choice Experiment Eliciting Relative Preferences for Equity and Efficiency. Value in Health 2012 May;15(3):534-9.
- Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res. 2012; 12: 454.
- Youngkong S, Baltussen R, Tantivess S, Mohara A, and Teerawattananon Y. Multi-criteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value in Health 2012 Sep-Oct;15(6):961-70.
- Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012 Feb 17;10.
- Baltussen R, Norheim O, Johri M. Fairness in service choice: an important yet underdeveloped path to universal coverage. Tropical Medicine and International Health. 2011 Jul;16(7):838-9.
- Chitama D, Baltussen R, Ketting E, Kamazima S, Nswilla A, Mujinja PG. From papers to practices: district level priority setting processes and criteria for family planning, maternal, newborn and child health interventions in Tanzania. BMC Womens Health. 2011 Oct 21;11:46.
- Youngkong S, Tromp N, Chitama D. The EVIDEM framework and its usefulness for priority setting across a broad range of health interventions. Cost Effectiveness and Resource Allocation. 2011;9(8).
- Baeten S, Baltussen R, Uyl-de Groot C, Bridges J, Niessen L. Incorporating equity-efficiency interactions in cost-effectiveness analysis-three approaches applied to breast cancer control. Value Health. 2010;13(5):573-9.
- Baltussen R, Youngkong S, Paolucci F, Niessen L. Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experience. Health Policy. 2010;96(3):262-4.
- Youngkong S, Baltussen R, Tantivess S, Koolman X, Teerawattananon Y. Criteria for priority setting of HIV/AIDS interventions in Thailand: A discrete choice experiment. BMC Health Service Research. 2010;10:197-206.
- Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies.Tropical Medicine and International Health. 2009;14(8):930-9.
Sorted on year and first author.
- Angelis A, Baltussen R, and Tervonen T. The Need for Novel Approaches in Assessing the Value of COVID-19 Vaccines. American Journal of Public Health 111, no. 2 (February 1, 2021): pp. 205-208.
- Mueller D, Gutierrez-Ibarluzea I, Chiumente M, Oortwijn W. Towards a common understanding of competencies for HTA: Enhancing educational and training programs around the globe. International Journal of Technology Assessment in Health Care, 2021 37, e29, 1–10.
- Baeroe K, Jansen M, Kerasidou A. The ethics of machine learning in healthcare: Is exceptionalism required after all? American journal of Bioethics 2020; 20: 48 - 51.
- Broekhuizen H, Lansu M, Gajewski J, Pittalis C, Ifeanyichi M, Juma A, Marealle P, Kataika E, Chilonga K, Rouwette E, Brugha R, Bijlmakers L. Using group model building to capture the complex dynamics of scaling up district-level surgery in Arusha region, Tanzania. Int J Health Pol Manag 2020:1-9.
- Gad M, Salem A, Oortwijn W, Hill R, Godman B. Mapping of current obstacles for rationalising use of medicines (CORUM) in Europe: Current situation and potential solutions. Front. Pharmacol. 2020;11:144.
- Gajewski J, Wallace M, Pittalis C, Mwapasa G, Borgstein E, Bijlmakers L, Brugha R. Why do they leave? Challenges to retention of surgical clinical officers in district hospitals in Malawi. Int J Health Pol Manag 2020, 1-8.
- Hangoma P, Bulawayo M, Chewe M, et al. The potential health and revenue effects of a tax on sugar sweetened beverages in Zambia. BMJ Global Health 2020;5:e001968.
- Heemskerken P, Broekhuizen H, Gajewski K, Brugha R, Bijlmakers L. Barriers to surgery performed by non-physician clinicians in sub-Saharan Africa – a scoping review. Human Res Health 2020; 18:51.
- Jakub Gajewski, Nasser Monzer, Chiara Pittalis, Leon Bijlmakers, Mweene Cheelo, John Kachimba, Ruairi Brugha. Supervision as a tool for building surgical capacity of district hospitals: the case of Zambia. Human Res Health, 2020,18:25.
- Mwapasa G, Pittalis C, Clarke M, Bijlmakers L, Le G, Mkandawire N, Brugha R, Borgstein E, Gajewski J. Evaluation of a managed surgical consultation network in Malawi. World J Surg 2020, 45:356-61.
- O’Rourke B, Oortwijn W, Schuller T. Announcing the New Definition of HTA. Letter to the Editor, Value in Health, 2020; 23(6):824-825
- O’Rourke B, Oortwijn W, Schuller T. The new definition of health technology assessment: a milestone in international collaboration. International Journal of Technology Assessment in Health Care, 2020 May 13;1-4.
- Pittalis C, Brugha R, Bijlmakers L, Mwapasa G, Borgstein E, Gajewski. Patterns, quality and appropriateness of surgical referrals in Malawi. Trop Med Int Health 2020, 25(7):824-33.
- Sacchini D, Refolo P et al. Core Competencies for Ethics Experts in HTA. Int J Technol Assess Health Care. 2020;36(6):534-539. Epub 2020 Dec 9.
- Wammes J, Frederix G, Govaert P, Determann D, Evers S, Paulus A, Stadhouders N, Jeurissen P, Oortwijn W, Adang E. Case-studies of displacement effects in Dutch hospital care. BMC Health Services Research, 20; 263 (2020).