Academic papers

• Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., ÓhÉigeartaigh, S., Korinek, A. and Anderljung, M., Open-Sourcing Highly Capable Foundation Models. Centre for the Governance of AI report.

• Trager, R., Harack, B., Reuel, A., Carnegie, A., Heim, L., Ho, L., Kreps, S., Lall, R., Larter, O., ÓhÉigeartaigh, S.S. and Staffell, S., 2023. International Governance of Civilian AI: A Jurisdictional Certification Approach. arXiv preprint arXiv:2308.15514.

• Chiodo, M., & Müller, D. (2023). Manifesto for the Responsible Development of Mathematical Works--A Tool for Practitioners and for Management. arXiv preprint arXiv:2306.09131.

• Richards, C. E., Tzachor, A., Avin, S., & Fenner, R. (2023). Rewards, risks and responsible deployment of artificial intelligence in water systems. Nature Water, 1-11.

• Anderljung, M., Barnhart, J., Leung, J., Korinek, A., O'Keefe, C., Whittlestone, J., …Avin, S.,... & Wolf, K. (2023). Frontier AI Regulation: Managing Emerging Risks to Public Safety. arXiv preprint arXiv:2307.03718.

• Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Avin, S., ... & Dafoe, A. (2023). Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324.

• Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Burden, J.,... & Maharaj, T. (2023, June). Harms from Increasingly Agentic Algorithmic Systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 651-666).

• Burnell, R., Schellaert, W., Burden, J., Ullman, T. D., Martinez-Plumed, F., Tenenbaum, J. B., ... & Hernandez-Orallo, J. (2023). Rethink reporting of evaluation results in AI. Science, 380(6641), 136-138.

• Schellaert, W., Martınez-Plumed, F., Vold, K., Burden, J., Casares, P. A., Loe, B. S., ...ÓhÉigeartaigh, S. S. & Hernández-Orallo, J. (2023). Your Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research, 77, 377-394.

• Burden, J., Hernandez-Orallo, J., & Heigeartaigh, S.S. (2022). How Sure to Be Safe? Difficulty, Confidence and Negative Side Effects. In NeurIPS ML Safety Workshop.

• Voudouris, K., Donnelly, N., Rutar, D., Burnell, R., Burden, J., Hernández-Orallo, J., & Cheke, L. G. (2022). Evaluating object permanence in embodied agents using the animal-AI environment. https://ceur-ws. org/Vol-3169/paper2.pdf

• Sundaram, L., Maas, M. M., & Beard, S. J. (2022). Seven Questions for Existential Risk Studies. Forthcoming in Managing Extreme Technological Risk (ed. Catherine Rhodes).

• Clarke, S., & Whittlestone, J. (2022). A Survey of the Potential Long-term Impacts of AI. AIES 2022

• Casares, P. A. M., Loe, B. S., Burden, J., ÓhÉigeartaigh, S. S. & Hernández-Orallo, J. (2022). How General-Purpose Is a Language Model? Usefulness and Safety with Human Prompters in the Wild. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 5).

• Srivastava, A., et al, including Burden, J. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.

• Clarke, S., & Whittlestone, J. (2022). AI Challenges for Society and Ethics. The Oxford Handbook of AI Governance.

• Maas, M., Lucero-Matteucci, K., Cooke, D. (2022). Military Artificial Intelligence as Contributor to Global Catastrophic Risk. CCCR 2020.

• Barnett, P., & Burden, J. (2022). Oases of Cooperation: An Empirical Evaluation of Reinforcement Learning in the Iterated Prisoner's Dilemma. In SafeAI@ AAAI.

• Tzachor, A., Devare, M., King, B., Avin, S., Ó hÉigeartaigh, S. S. (2022) Responsible artificial intelligence in agriculture requires systemic understanding of risks and externalities. Nature Machine Intelligence, 4(2), pp. 104-109.

• Gruetzemacher, R., & Whittlestone, J. (2022) The transformative potential of artificial intelligence. Futures, 135.

• Whittlestone, J. (2021). AI and Decision-Making. Future Morality, (Oxford University Press), 102.

• Avin, S., Belfield, H., Brundage, M., Krueger, G., Wang, J., Weller, A., Anderljung, M., Krawczuk, I., Krueger, D., Lebensold, J., Maharaj, T., Zilberman, N. (2021) Filling gaps in trustworthy development of AI. Science, 374(6573), pp.1327-1329.

• Hua, S., & Belfield, H. (2021) AI & Antitrust: Reconciling Tensions Between Competition Law and Cooperative AI Development. Yale Journal of Law & Technology, 23.

• Hernandez-Orallo, J., Sheng Loe, B., Cheke, L., Martinez-Plumed, F., Ó hÉigeartaigh, S. S. (2021) General intelligence disentangled via a generality metric for natural and artificial intelligence. Scientific Reports, 11.

• Whittlestone, J., & Clark, J. (2021) Why and How Governments Should Monitor AI Development. https://arxiv.org/pdf/2108.12427.pdf.

• Maas, M. (2021) Aligning AI Regulation to Sociotechnical Change. In: Justin Bullock, Baobao Zhang, Yu-Che Chen, Johannes Himmelreich, Matthew Young, Antonin Korinek & Valerie Hudson (eds.). Oxford Handbook on AI Governance (Oxford University Press, 2022 forthcoming).

• Martinez-Plumed, F., Barredo, P., Ó hÉigeartaigh, S. S., Hernandez-Orallo, J. (2021) Research community dynamics behind popular AI benchmarks. Natural Machine Intelligence 3, 581–589.

• Maas, M. (2021) AI, Governance Displacement, and the (De)Fragmentation of International Law. ISA 2021.

• Cave, S., Whittlestone, J., Nyrup, R., Ó hÉigeartaigh, S. S., and Calvo, R. (2021). Using AI ethically to tackle covid-19. BMJ, 372.

• Whittlestone, J., Arulkumaran, K., & Crosby, M. (2021). The Societal Implications of Deep Reinforcement Learning. Journal of Artificial Intelligence Research, 70, 1003-1030.

• Cremer, C. Z., & Whittlestone, J. (2021). Artificial Canaries: Early Warning Signs for Anticipatory and Democratic Governance of AI. International Journal of Interactive Multimedia & Artificial Intelligence, 6(5).

• Kunz, M., & Ó hÉigeartaigh, S. (Forthcoming) Artificial Intelligence and Robotization. In Robin Geiß and Nils Melzer (eds.), Oxford Handbook on the International Law of Global Security (Oxford University Press, Forthcoming) (2021).

• Liu, H. Y., & Maas, M. M. (2021). ‘Solving for X?’Towards a problem-finding framework to ground long-term governance strategies for artificial intelligence. Futures, 126, 102672.

• Burden, J., & Hernández-Orallo, J. (2021, January). Negative Side Effects and AI Agent Indicators: Experiments in SafeLife. In SafeAI@ AAAI.

• Stix, C., & Maas, M. M. (2020). Bridging the gap: the case for an ‘Incompletely Theorized Agreement’ on AI policy. AI and Ethics, 1-11.

• Cihon, P., Maas, M. M., & Kemp, L. (2020). Fragmentation and the Future: Investigating Architectures for International AI Governance. Global Policy, 11(5), 545-556.

• Cremer, C. Z., & Whittlestone, J. (2020). Canaries in Technology Mines: Warning Signs of Transformative Progress in AI. Evaluating Progress in AI workshop, ECAI.

  • Presents a methodology for identifying early warning signs of societally-impactful progress in AI.

• Barredo, P., Hernandez-Orallo, J., Martinez-Plumed, F. & O hEigeartaigh, S.S. (2020) The Scientometrics of AI Benchmarks: Unveiling the Underlying Mechanics of AI Research. Evaluating Progress in AI Workshop, ECAI 2020

  • Examines the research and collaboration dynamics underpinning progress on key benchmark challenges in AI (e.g. image dataset analysis challenges such as Imagenet). These benchmark challenges have played a key role in shaping and guiding progress in AI in recent years.

• Tzachor, A., Whittlestone, J., Sundaram, L. & Ó hÉigeartaigh, S.S. (2020) Artificial intelligence in a crisis needs ethics with urgency. Nature Machine Intelligence

• ÓhÉigeartaigh, S.S., Whittlestone, J., Liu, Y. et al. (2020). Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and Governance. Philosophy & Technology

  • Explores the basis for mistrust and adversarial rhetoric between global AI-leading regions, identifies points of common agreement on AI ethics and governance, and makes recommendations for improving mutual understanding and cooperation.

• Hernández-Orallo, J., Martınez-Plumed, F., Avin, S., Whittlestone, J., & Ó hÉigeartaigh, S.S. (2020) AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues. ECAI 2020

  • Maps research into AI safety and risk challenges (e.g. adversarial attacks, reliability and robustness, value alignment) to the literature on different research directions in AI progress, as an early step in developing a better understanding of the types of safety challenges that may be associated with different paradigms of AI scientific progress.

• Shackelford, G. E., Kemp, L., Rhodes, C., Sundaram, L., ÓhÉigeartaigh, S. S., Beard, S., ... & Jones, E. M. (2020). Accumulating evidence using crowdsourcing and machine learning: A living bibliography about existential risk and global catastrophic risk. Futures, 116, 102508..

• Rich, A.S., Rudin, C., Jacoby, D.M.P., Ó hÉigeartaigh, S.S., Cave, S., Dihal, K. et al. (2020) AI reflections in 2019. Nature Machine Intelligence 2, 2–9

• Belfield, Haydn. (2020). Activism by the AI Community: Analysing Recent Achievements and Future Prospects. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 15-21).

• Avin, S., Gruetzemacher, R., & Fox, J. (2020). Exploring AI Futures Through Role Play. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 8-14).

• Prunkl, C., & Whittlestone, J. (2020). Beyond Near-and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 138-143).

• Cihon, P., Maas, M. M., & Kemp, L. (2020). Should Artificial Intelligence Governance be Centralised? Design Lessons from History. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 228-234).

• Burden, J., & Hernández-Orallo, J. (2020, January). Exploring AI safety in degrees: Generality, capability and control. In SafeAI@ AAAI.

• Hagerty, A., & Rubinov, I. (2019). Global AI ethics: a review of the social impacts and ethical implications of artificial intelligence. arXiv preprint arXiv:1907.07892.

• Whittlestone, J., & Ovadya, A. (2019). The tension between openness and prudence in responsible AI research. NeurIPS 2019 Joint Workshop on AI for Social Good.

• Ovadya, A., & Whittlestone, J. (2019). Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. arXiv preprint arXiv:1907.11274.

• Vold, K., & Whittlestone, J. (2019). Privacy, Autonomy, and Personalised Targeting: rethinking how personal data is used. IE Report on Data, Privacy and the Individual.

• Hernández-Orallo, J., Martínez-Plumed, F., Avin, S. and Ó hÉigeartaigh, S.S. (2019). Surveying Safety-relevant AI Characteristics. AAAI 2019.

  • Short-listed for best paper at the AAAI 2019 SafeAI workshop. The paper presents a framework of characteristics of AI systems with relevance to technical safe design, and analyses AI safety problems being explored in current research agendas within this framework.

• Whittlestone, J., Nyrup, R., Alexandrova, A., & Cave, S. (2019). The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions. In Proceedings of the AAAI/ACM Conference on AI Ethics and Society, Honolulu, HI, USA (pp. 27-28).

  • Highlights the limitations of ethical and societal principles in guiding AI development; identifies and discusses ways in which tensions emerge between principles when we attempt to implement principles in the real world.

• Cave, S., & Ó hÉigeartaigh, S. S. (2019). Bridging near-and long-term concerns about AI. Nature Machine Intelligence, 1(1), 5.

  • Argues that divisions in the AI impacts field between ‘present day’ and ‘longer-term’ concerns risks missing out valuable insights; many AI ethics challenges will evolve quickly and a more foresight-oriented approach is needed.

• Avin, S. (2019). Exploring artificial intelligence futures. Journal of AI Humanities.

  • Presents a range of tools for artificial intelligence futures work, calling for more work in data-driven, realistic, integrative, and participatory scenario role-plays.

• Avin, S. & Amadae, S.M. (2019). Autonomy and machine learning at the interface of nuclear weapons, computers and people. In The Impact of Artificial Intelligence on Strategic Stability and Nuclear Risk, SIPRI, (pp. 105-118).

  • Analyses emerging AI and cyber-related risks as nuclear weapons systems are updated and digitised.

• Martinez-Plumed, F., Avin, S., Brundage, M., Dafoe, A., Ó hÉigeartaigh, S.S., & Hernández-Orallo, J. (2018). Accounting for the neglected dimensions of AI progress. arXiv preprint arXiv:1806.00610.

Reports

• Clarke, S., Whittlestone, J., Maas, M., Belfield, H., Hernandez-Orallo, J., Ó hÉigeartaigh, S.S. (2021) Submission of Feedback to the European Commission’s Proposal for a Regulation laying down harmonised rules on artificial intelligence. CSER.

• Ord, T., Mercer, A., Dannreuther, S., Belfied, H., Whittlestone, J., Leung, J., Anderljung, A., Nelson, C., Lewis, G., Millett, P., Hilton, S. (2021) Future Proof: the opportunity to transform the UK's resilience to extreme risks. Centre for Long-Term Resilience.

• Seger, E., Avin, S., Pearson, G., Briers, M., Ó Heigeartaigh, S., Bacon, H. (2020). Tackling threats to informed decision- making in democratic societies. Alan Turing Institute.

• Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Ó hÉigeartaigh, S.S. ... & Maharaj, T. (2020). Toward trustworthy AI development: mechanisms for supporting verifiable claims. arXiv preprint arXiv:2004.07213.

• Stix, Charlotte (2019). A Survey of the European Union’s Artificial Intelligence Ecosystem.

  • Covers recent developments in European Union regulatory, ethics, and funding initiatives relating to AI.

• Whittlestone, J., Nyrup, R., Alexandrova, A., Dihal, K., & Cave, S. (2019). Ethical and societal implications of algorithms, data, and artificial intelligence: a roadmap for research. London: Nuffield Foundation.

• Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., ... & Anderson, H. (2018). The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228.

  • Collaboration between CSER/AI:FAR researchers (Shahar Avin, Sean O hEigeartaigh, Haydn Belfield, Simon Beard), OpenAI, FHI, and CNAS. Emerging threats posed by AI in physical, digital and political security. 

Policy submissions

• Whittlestone, J., Edgerton Avin, S., Clark, J., & Mueller, J. (2022). Future of compute review-submission of evidence. 

• Belfield, H., Maas, M., Avin, S., & Ó hÉigeartaigh, S. S. (2022). Response to NIST AI Risk Management Framework Concept Paper.

• Avin, S., Sundaram, L., Whittlestone, J., Maas, M.M., Hobson, T. (2021). Submission of Evidence to The House of Lords Select Committee on Risk Assessment and Risk Planning.

• Belfield H., Hernandez-Orallo, J., O hEigeartaigh, S., Maas, M., Hagerty, A., Whittlestone, J. (2020). Response to the European Commission’s Whitepaper on AI.

• Belfield, H., Jayanti, A., and Avin, S. (2020). Defence Industrial Policy: Procurement and prosperity. Written Evidence to the UK Parliament Defence Committee's Inquiry on Defence Industrial Policy

• Whittlestone, J., Vold, K. and Alexandrova, A. (2019). The potential harms of online targeting. Evidence submitted to the UK Centre for Data Ethics and Innovation.

• Nyrup, R., Whittlestone, J. and Cave, S. (2019). Why Value Judgements Should Not Be Automated. Evidence submitted to the Committee on Standards in Public Life.

• Kemp, L., Cihon, P., Maas, M., Belfield, H., Ó hÉigeartaigh, S., Leung, J. & Cremer, C.Z.  UN High-level Panel on Digital Cooperation: A Proposal for International AI Governance  (2019).

  • Chosen as one of 6 (out of 108) submissions to be presented at a UN town hall meeting on AI governance; AI:FAR’s Haydn Belfield presented.

• Belfield, H. & Avin, S. Response to the European Commission’s High-Level Expert Group on Artificial Intelligence request for evidence. (2019).

  • Provided recommendations (based on previous research) on technical and governance measures for trustworthy and robust AI based on the Malicious Use of AI report.

• Response to the US National Institute of Standards and Technology request for information on standards for artificial intelligence (2019) (Ó hÉigeartaigh & Belfield, with colleagues from collaborating institutions)

Essays and Blog Posts

• Belfield, H., & Hua, S. S. (2022). Compute and Antitrust: Regulatory implications of the AI hardware supply chain, from chip design to cloud APIs. Verfassungsblog.

• Maas, M. (2022) Paths Untaken: The History, Epistemology and Strategy of Technological Restraint, and lessons for AI. (online) Verfassungsblog.

• Belfield, H., & Hua, S. (2022) Effective Enforceability of EU Competition Law Under Different AI Development Scenarios. [online] Verfassungsblog.

• Maas, M. (2022) Strategic Perspectives on Long-term AI Governance: Introduction. (online) Effective Alturism Forum.

• Belfield, H. (2022) Are you really in a race? The Cautionary Tales of Szilárd and Ellsberg. (online) Effective Alturism Forum.

• Belfield, H., & Rhul, C. (2022) Why policy makers should beware claims of new ‘arms races’. (online) Bulletin of the Atomic Scientists.

• Clarke, S. (2022) The longtermist AI governance landscape: a basic overview. (online) Effective Alturism Forum.

• Clake, S. (2022) Clarifications about structural risk from AI. (online) Effective Alturism Forum.

• Clarke, S., & Martin, S. (2021) Distinguishing AI takeover scenarios. (online) AI Alignment Forum.

• Clarke, S., Carlier, A., Schuett, J. (2021) Survey on AI existential risk scenarios. (online) AI Alignment Forum.

Other Resources

• Global AI Governance: Overview of international law governing applications of AI and related technologies by former AI:FAR researcher Martina Kunz www.globalAIgov.org