Paper Abstracts

Evaluation Framework for Large Language Model-based Conversational Agents

The integration of Large Language Models (LLM) in Conversational Agents (CA) enables a significant advancement in the agents’ ability to understand and respond to user queries in a more human-like manner. Despite the widespread adoption of LLMs in these agents, there exists a noticeable lack of research on standardized evaluation methods. Addressing this research gap, our study proposes a comprehensive evaluation framework tailored explicitly to LLM-based conversational agents. In a Design Science Research (DSR) project, we construct an evaluation framework that incorporates four essential components: the pre-defined objectives of the agents, corresponding tasks, and the selection of appropriate datasets and metrics. Our framework outlines how these elements relate to each other in the evaluation and enables a structured approach for the evaluation. We demonstrate how such a framework enables a more systematic evaluation process. This framework can be a guiding tool for researchers and developers working with LLM-based conversational agents.

Designing for High Availability – A Reference Architecture for IoT Data Platforms

This paper presents a novel Reference Architecture (RA) for designing complex Internet of Things (IoT) data platforms with a focus on High Availability (HA). HA is crucial in various IoT applications, from smart cities to industrial sectors like smart farming, where system disruptions can lead to significant losses. The RA aims to guide business-es in designing IoT data platforms that ensure uninterrupted operations and maintain data integrity. We identify essential features such as redundancy, replication, and a layered approach by synthesizing insights from existing RAs and architectures of IoT data platforms, particularly in large-scale implementations. A Design Science Research approach is used to develop a comprehensive RA for HA iteratively. This comprehensive framework contributes to the field of Information Systems by providing a blueprint for developing scalable and resilient IoT data platforms capable of accommodating the rapid growth of IoT data across diverse domains.

Enabling Smart Collaboration Spaces in Organizations: Foundations and an Integrated IoT Architecture

This research explores the foundational aspects of Smart Collaboration Spaces (SCS), which integrate environmental data such as air quality and workplace occupancy detection to enhance collaboration and productivity within organizations. The study begins with a literature review of SCS foundations, examining existing technologies and frameworks. Building upon this foundation, an integrated Internet of Things (IoT) architecture for SCS is proposed, considering the edge-fog-cloud continuum and incorporating various characteristics of SCS. The development of this architecture is facilitated by utilizing a university campus as a laboratory setting, allowing for real-world testing and validation. The primary IoT infrastructure employed in this architecture leverages Long-Range Wide Area Network (LoRaWAN) technology, including LoRa Gateways, IoT sensors, and the university's own Infrastructure as a Service (IaaS) cloud. Through this research, insights, and an understanding of SCS is provided, along with an integrated IoT architecture which may be leveraged for implementing SCS in organizational settings.