BenchCouncil Transactions on Benchmarks, Standards and Evaluations

BenchCouncil Transactions on Benchmarks, Standards and Evaluations (TBench) is an open-access journal dedicated to advancing the field of benchmarks, data sets, standards, evaluations and optimizations.

We invite submissions covering a wide range of topics from various disciplines, with a particular emphasis on interdisciplinary research. Whether it pertains to computers, AI, medicine, education, finance, business, psychology, or other social disciplines, all relevant contributions are welcome.

At TBench, we prioritize the reproducibility of research. We strongly encourage authors to ensure that their articles are prepared for open-source or artifact evaluation before submission.

Areas of interest include, but are not limited to:

Problem definition as a benchmark

Position articles on the definitions of emerging or future problems/ challenges

Position articles on opening new research areas

Position articles on investigating the impact of new technologies on different disciplines

Research articles on instantiating new problem settings

Evaluation standard as a benchmark

Definition, design, implementation, and validation of evaluation standards

Evaluation methodology and metrics

New abstractions and tools in evaluation

Simulation, emulation, and testbed methodologies and systems in evaluation

Industry best practice as a benchmark

Searching and summarizing industry best practice

Evaluation and optimization of industry practice

Retrospective of industry practice

Characterizing and optimizing real-world applications and systems

Benchmarking

State-of-the-art solution as a benchmark

State-of-the-art solutions to well-known benchmarks

Benchmarking state-of-the-art solutions

Preliminary but insightful solutions to new or emerging problems or benchmarks

Evaluations of state-of-the-art solutions in the real-world setting

Data set as a benchmark

Explicit or implicit problem definition deduced from the data set

Detailed descriptions of research or industry datasets, including the methods used to collect the data and technical analyses supporting the quality of the measurements

Analyses or meta-analyses of existing data

Systems, technologies, and techniques that advance data sharing and reuse to support reproducible research

Tools that generate large-scale data while preserving their original characteristics

Evaluating the rigor and quality of the experiments used to generate the data and the completeness of the data description

Workload characterization, evaluation, and retrospective of design/implementation of real-world:

Computer or AI applications or systems

Finance applications or systems

Education applications or systems

Business applications or systems

Medicine applications or systems

Other industry applications or systems

Measurement and evaluation:

Instrumentation, sampling, tracing, and profiling of large-scale, real-world applications and systems

Collection and analysis of measurement data that yield new insights

Measurement-based modeling (e.g., workloads, scaling behavior, and assessment of performance bottlenecks)

Methods and tools to monitor and visualize measurement and evaluation data

Systems and algorithms that build on measurement-based findings

Advances in data collection, analysis, and storage, e.g., anonymization, querying, and sharing

Reappraisal of previous empirical measurements and measurement-based conclusions

Descriptions of challenges and future directions that the measurement and evaluation community should pursue

Methodologies, metrics, abstractions, algorithms, and tools for:

Analytical modeling techniques and model validation

Workload characterization and benchmarking

Performance, scalability, power, and reliability analysis

Sustainability analysis and power management

System measurement, performance monitoring, and forecasting

Anomaly detection, problem diagnosis, and troubleshooting

Capacity planning, resource allocation, run-time management, and scheduling

Experimental design, statistical analysis, and simulation