AAAI HCOMP 2024 

Responsible Crowd Work for Better AI

October 16-19, 2024: HCOMP 2024 Pittsburgh, PA, USA

Important Dates

All times are 23.59 AoE

• June 5, 2024: Abstract submission (required to submit a full paper)
• June 12, 2024: Full papers due
• July 31, 2024: Notifications
• October 16-19, 2024: HCOMP 2024 Pittsburgh, PA, USA

Submission Guidelines

Submit to HCOMP using Easychair here.

Topics:

AAAI HCOMP is the premier venue for disseminating the latest research findings on human computation and crowdsourcing. Its focus is on research and practice into frameworks, methods, and systems that bring together people and machine intelligence to achieve better results. HCOMP 2024 will be held as an in-person conference in Pittsburgh, Pennsylvania from October 16-19. 

While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies, HCI design, social computing, machine learning, natural language processing, the broader realms of artificial intelligence (including LLMs and generative AI), economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice. 

With the unprecedented proliferation of AI systems across all domains and the adoption of machine learning across disciplines, there is a renewed focus on how complex AI systems are built, machine learning models are trained, and relevant data pipelines are set up to ensure responsible practices throughout these lifecycles. The role of human input and intelligence is being widely discussed in the age of LLMs, and generative AI. Human input serves a multitude of important purposes in these contexts, ranging from generation of training data to validation, evaluation, and facilitating oversight. Ensuring that data work is carried out in a fair, ethical, unbiased, and responsible manner every step of the way can help create better AI systems. It is in this spirit that the AAAI HCOMP 2024 theme focuses on “Responsible Crowd Work for Better AI.”

Topics of interest include:

• Methods to efficiently and effectively collect and use human feedback to build Generative AI models (e.g., pairwise judgments for RLHF, PPO, DPO, or statements for Constitutional AI).
• Humans versus LLMs for data annotation (assessment of quality, trade-offs, biases, evaluation, etc.); the use of LLMs for crowd work (hybrid workflows, productivity and work efficiency, quality, biases, oversight, etc.)
• Trust and reliance of crowd workers, data annotators, and other data work experts on LLM-based or generative AI tools
• Approaches to make crowd science FAIR (Findable, Accessible, Interoperable, Reproducible) and studies assessing and commenting on the FAIRness of human computation and crowdsourcing practice, replicability of crowdsourcing, and human computation experiments.
• Techniques that enable and enhance human-in-the-loop systems, making them more efficient, accurate, and human-friendly, including task design, quality assurance, answer inference, biases and subjectivity, incentives, gamification, task allocation, complex workflows, real-time crowdsourcing, etc.
• Methods, frameworks, techniques, and tools to help build appropriate reliance of humans on AI systems
• Studies about how people perform tasks individually, in groups, or as a crowd, including those drawing on techniques from human-computer interaction, social computing, computer-supported cooperative work, design, cognitive sciences, behavioral sciences, economics, etc.
• Data quality aspects of human-annotated and -curated datasets.
• Human computation and crowdsourcing to build people-centric AI systems and applications, including topics such as explainability and interpretability.
• Fairness, accountability, transparency, ethics, and policy implications for crowdsourcing and human computation.
• Studies about how people and intelligent systems interact and collaborate, coordinate, or compete, and studies exploring the influences and impact of intelligent systems on society.
• Crowdsourcing applications and techniques, including but not limited to citizen science, collective action, collective intelligence, the wisdom of crowds, crowdsourcing contests, crowd creativity, crowdfunding, paid microtasks, crowd ideation, crowd sensing, and prediction markets.
• Studies that inform our understanding of the future of work, distributed work, the freelancer economy, open innovation, and citizen-led innovation.

Submissions may, therefore, cover theory, user studies, tools, and applications that present novel, interesting, impactful interactions between people and computational systems. These cover a broad range of scenarios, from classical human computation, the wisdom of crowds, and all forms of crowdsourcing to people-centric AI methods, systems, and applications.

Format:

Authors are invited to submit anonymized full papers of variable length up to a maximum of 8 pages (including all content, figures, and tables). Additional pages may contain references only (i.e., max. 8 pages + references). Shorter and focussed submissions are welcome and reviewers will assess the contributions of the work accordingly. Papers must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2023 Author Kit for details (available templates: AAAI 2023 Author Kit on Overleaf or AAAI 2024 Author Kit.zip [Word | LaTeX]). The AAAI copyright block is not required on submissions but must be included in the final accepted versions. 

Authors are invited, but not required, to include supplemental materials such as executables and data files so that reviewers can reproduce results in the paper, images, additional videos, related papers, more detailed explanations, derivations, or results. These materials will be viewed only at the reviewers’ discretion, who are only obligated to read the submitted papers.

Accepted full papers will be published in the HCOMP conference proceedings and included in the AAAI Digital Library. AAAI enables authors to use Open Responsible AI Licenses (Open RAIL), licenses designed to permit free and open access, re-use, and downstream distribution of derivatives of AI artifacts as long as the behavioral-use restrictions always apply (including to derivative works). Need Help Deciding if an AI Pubs License is Right for You? Take a look here:  https://www.licenses.ai/blog/2023/3/3/ai-pubs-rail-licenses. Please consider this as a way to share your work with the community. If you have questions, comments, or feedback, please contact the RAIL team here: https://www.licenses.ai/contact.

If your paper is accepted, you will be required to present it in person at the AAAI HCOMP 2024 conference in Pittsburgh. At least one author of each accepted paper must register for the main conference to present the work or acceptance will be withdrawn. 

Double Submission Policy:

Papers submitted to the HCOMP conference must represent original work that has not been previously published or that is not under simultaneous peer review for any other peer-reviewed archival conference or journal. Note that –

• Papers that have appeared at a conference with published proceedings constitute previously published work.
• Papers that overlap with other papers that have appeared at a conference with published proceedings must contain significant new results.
• Papers that have appeared at a workshop do not constitute previously published work, as long as the paper submitted to HCOMP is an extension of the workshop paper. Extensions might include new results, more in-depth analysis, an evaluation that was not part of the workshop paper, or further experiments.

Reviews

HCOMP 2024 will adopt a double-blind review process. Submissions should omit any author names, affiliations, or other identifying information. Submissions not complying with this guidance will be desk-rejected.

Each paper will be reviewed by at least three members of the program committee and one SPC member. Reviewers will be instructed to evaluate submissions according to specific review criteria. We encourage authors to review them before submission.

HCOMP 2024 will recognize one best paper, one best student paper, and up to two runner-ups. Reviewers will be asked to flag papers they deem worthy of a prize. The general chairs will set up a small committee that will read the nominated papers, consider the comments of the reviewers, and assess the presentation to determine the winners.

Contact

All questions about submissions should be emailed to hcomp24 [AT] easychair [DOT] org.