Cleanup repository & Fix BBB GoldStandard

README.md

@@ -1,5 +1,9 @@
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6966831.svg)](https://doi.org/10.5281/zenodo.6966831)
 
+> [!TIP]
+> **This is not the version of the benchmark from the original ECSA 2022 paper. The original version is located at: https://doi.org/10.5281/zenodo.6966832**
+> We also removed a lot of files that were not used anymore. See [#20](https://github.com/ardoco/benchmark/pull/20).
+
 # Benchmark
 This repository contains a benchmark for traceability link recovery (TLR) between textual Software Architecture Documentation (SAD) and Software Architecture Models (SAM).
 It was initially published in the paper [Establishing a Benchmark Dataset for Traceability Link Recovery Between Software Architecture Documentation and Models](https://doi.org/10.1007/978-3-031-36889-9_30).
@@ -13,19 +17,21 @@ Each project of the benchmark is structured as follows:
 	* The folder `code` contains a code model. The version of the code is stated in a README.md next to the model. The model is an `ArDoCo Code Model`. The model can be loaded using the [ArDoCo Code Extractor](https://github.com/ArDoCo/Core/blob/main/stages/model-provider/src/main/java/edu/kit/kastel/mcse/ardoco/core/models/connectors/generators/code/CodeExtractor.java#L47).
 * The folder `text_<year>` contains a documentation of the project.
 	* The text file(s) in the project folder contains the SAD of the projects as plain text.
-* The folder `diagrams_<year>` contains the informal diagrams of the project.
+	* If available: contains the informal diagrams of the project.
 * The folder `goldstandards` contains all gold standards for the project. In the following, the `<year>` refers to the year of the artifact. Additional classifiers are added to the end of the file names. You will find more information about these classifiers in the README of the respective project artifacts.
 	* The file `goldstandard_sad_<year>-sam_<year>.csv` contains the traceability links between SAD and SAM. It links the model elements by id with the sentences by their number 
 	* The file `goldstandard_sad_<year>-sam_<year>_UME.csv` contains all IDs of model elements that are contained in the model but not described in the text.
 	* The file `goldstandard_sad_<year>_code_<year>.csv` contains the traceability links between SAD and code models.
 	* The file `goldstandard_sam_<year>-code_<year>.csv` is a gold standard for mapping the architecture elements and code elements.
-	* The file `goldstandard_sad_id_<year>.json` contains the traceability links between SAD and informal diagrams. The JSON schema is specified in the [DiagramSchema](DiagramSchema.json) file.
+	* The file `goldstandard_sad_diagram_<year>.json` contains the traceability links between SAD and informal diagrams. The JSON schema is specified in the [DiagramSchema](DiagramSchema.json) file.
 
 
 
 ## Using the benchmark
 In order to provide an easy approach to use the benchmark, we provide an example TLR approach called [Simple Tracelink Discovery (STD)](https://github.com/ArDoCo/SimpleTracelinkDiscovery/) that uses this benchmark in its [evaluation](https://github.com/ArDoCo/SimpleTracelinkDiscovery/tree/main/src/test/java/io/github/ardoco/simpletracelinkdiscovery/eval).
-Therefore, the benchmark is linked to the STD repository via a [git subtree](https://github.com/ArDoCo/SimpleTracelinkDiscovery/tree/main/src/test/resources/benchmark).
+
+We suggest using *git subtree* to include the benchmark in your project:
+`git subtree add --prefix destination-folder git@github.com:ardoco/benchmark.git <<VERSION>>`
 
 ## Projects
 

bigbluebutton/goldstandards/goldstandard_sad_2021-code_2023.csv

@@ -27,113 +27,107 @@ sentenceID,codeID
 20,build/packages-template/bbb-html5/
 21,bigbluebutton-html5/
 21,build/packages-template/bbb-html5/
-25,akka-bbb-apps/
-25,build/packages-template/bbb-apps-akka/
-28,bbb-common-web/src/main/java/org/bigbluebutton/web/
-28,bigbluebutton-web/
-28,build/packages-template/bbb-web/
-34,bbb-common-web/src/main/java/org/bigbluebutton/web/
-34,bigbluebutton-web/
-34,build/packages-template/bbb-web/
-35,bbb-common-web/src/main/java/org/bigbluebutton/web/
-35,bigbluebutton-web/
-35,build/packages-template/bbb-web/
+26,akka-bbb-apps/
+26,build/packages-template/bbb-apps-akka/
+30,bbb-common-web/src/main/java/org/bigbluebutton/web/
+30,bigbluebutton-web/
+30,build/packages-template/bbb-web/
 36,bbb-common-web/src/main/java/org/bigbluebutton/web/
 36,bigbluebutton-web/
 36,build/packages-template/bbb-web/
-37,bigbluebutton-html5/
-37,build/packages-template/bbb-html5/
-44,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
-45,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
-45,bigbluebutton-html5/
-45,build/packages-template/bbb-html5/
-46,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/Keys.java
-46,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisAwareCommunicator.java
-46,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisStorageService.java
-47,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/Keys.java
-47,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisAwareCommunicator.java
-47,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisStorageService.java
-49,akka-bbb-apps/
-49,build/packages-template/bbb-apps-akka/
-50,akka-bbb-apps/
-50,build/packages-template/bbb-apps-akka/
+37,bbb-common-web/src/main/java/org/bigbluebutton/web/
+37,bigbluebutton-web/
+37,build/packages-template/bbb-web/
+38,bbb-common-web/src/main/java/org/bigbluebutton/web/
+38,bigbluebutton-web/
+38,build/packages-template/bbb-web/
+39,bigbluebutton-html5/
+39,build/packages-template/bbb-html5/
+46,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
+47,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
+47,bigbluebutton-html5/
+47,build/packages-template/bbb-html5/
+48,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/Keys.java
+48,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisAwareCommunicator.java
+48,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisStorageService.java
+49,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/Keys.java
+49,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisAwareCommunicator.java
+49,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/RedisStorageService.java
 51,akka-bbb-apps/
 51,build/packages-template/bbb-apps-akka/
 52,akka-bbb-apps/
 52,build/packages-template/bbb-apps-akka/
-55,akka-bbb-fsesl/
-55,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
-55,bbb-fsesl-client/
-55,bbb-fsesl-client/src/main/java/org/freeswitch/
-55,bbb-fsesl-client/src/test/java/org/freeswitch/
-55,build/packages-template/bbb-freeswitch-core/
-55,build/packages-template/bbb-freeswitch-sounds/
-55,build/packages-template/bbb-fsesl-akka/
-55,freeswitch.placeholder.sh
-56,akka-bbb-fsesl/
-56,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
-56,bbb-fsesl-client/
-56,bbb-fsesl-client/src/main/java/org/freeswitch/
-56,bbb-fsesl-client/src/test/java/org/freeswitch/
-56,build/packages-template/bbb-freeswitch-core/
-56,build/packages-template/bbb-freeswitch-sounds/
-56,build/packages-template/bbb-fsesl-akka/
-56,freeswitch.placeholder.sh
-58,akka-bbb-apps/
+53,akka-bbb-apps/
+53,build/packages-template/bbb-apps-akka/
+54,akka-bbb-apps/
+54,build/packages-template/bbb-apps-akka/
+57,akka-bbb-fsesl/
+57,bbb-fsesl-client/
+57,build/packages-template/bbb-fsesl-akka/
 58,akka-bbb-fsesl/
-58,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
+58,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
 58,bbb-fsesl-client/
-58,build/packages-template/bbb-apps-akka/
+58,bbb-fsesl-client/src/main/java/org/freeswitch/
+58,bbb-fsesl-client/src/test/java/org/freeswitch/
+58,build/packages-template/bbb-freeswitch-core/
+58,build/packages-template/bbb-freeswitch-sounds/
 58,build/packages-template/bbb-fsesl-akka/
-59,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
-59,bbb-fsesl-client/src/main/java/org/freeswitch/
-59,bbb-fsesl-client/src/test/java/org/freeswitch/
-59,build/packages-template/bbb-freeswitch-core/
-59,build/packages-template/bbb-freeswitch-sounds/
-59,freeswitch.placeholder.sh
-60,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
-60,bbb-fsesl-client/src/main/java/org/freeswitch/
-60,bbb-fsesl-client/src/test/java/org/freeswitch/
-60,build/packages-template/bbb-freeswitch-core/
-60,build/packages-template/bbb-freeswitch-sounds/
-60,freeswitch.placeholder.sh
+58,freeswitch.placeholder.sh
+60,akka-bbb-apps/
+60,akka-bbb-fsesl/
+60,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
+60,bbb-fsesl-client/
+60,build/packages-template/bbb-apps-akka/
+60,build/packages-template/bbb-fsesl-akka/
 61,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
 61,bbb-fsesl-client/src/main/java/org/freeswitch/
 61,bbb-fsesl-client/src/test/java/org/freeswitch/
 61,build/packages-template/bbb-freeswitch-core/
 61,build/packages-template/bbb-freeswitch-sounds/
 61,freeswitch.placeholder.sh
-63,bbb-webrtc-sfu.placeholder.sh
-63,build/packages-template/bbb-webrtc-sfu/
-64,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
-64,bbb-fsesl-client/src/main/java/org/freeswitch/
-64,bbb-fsesl-client/src/test/java/org/freeswitch/
-64,build/packages-template/bbb-freeswitch-core/
-64,build/packages-template/bbb-freeswitch-sounds/
-64,freeswitch.placeholder.sh
+62,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
+62,bbb-fsesl-client/src/main/java/org/freeswitch/
+62,bbb-fsesl-client/src/test/java/org/freeswitch/
+62,build/packages-template/bbb-freeswitch-core/
+62,build/packages-template/bbb-freeswitch-sounds/
+62,freeswitch.placeholder.sh
+63,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
+63,bbb-fsesl-client/src/main/java/org/freeswitch/
+63,bbb-fsesl-client/src/test/java/org/freeswitch/
+63,build/packages-template/bbb-freeswitch-core/
+63,build/packages-template/bbb-freeswitch-sounds/
+63,freeswitch.placeholder.sh
 65,bbb-webrtc-sfu.placeholder.sh
 65,build/packages-template/bbb-webrtc-sfu/
-68,bbb-webrtc-sfu.placeholder.sh
-68,build/packages-template/bbb-webrtc-sfu/
-70,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
-70,bbb-fsesl-client/src/main/java/org/freeswitch/
-70,bbb-fsesl-client/src/test/java/org/freeswitch/
-70,bigbluebutton-html5/
-70,build/packages-template/bbb-freeswitch-core/
-70,build/packages-template/bbb-freeswitch-sounds/
-70,build/packages-template/bbb-html5/
-70,freeswitch.placeholder.sh
-71,bbb-webrtc-sfu.placeholder.sh
-71,bigbluebutton-html5/
-71,build/packages-template/bbb-html5/
-71,build/packages-template/bbb-webrtc-sfu/
-74,bigbluebutton-html5/
-74,build/packages-template/bbb-html5/
-76,bbb-common-web/src/main/java/org/bigbluebutton/web/
-76,bigbluebutton-web/
-76,build/packages-template/bbb-web/
-77,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
-77,bigbluebutton-html5/
-77,build/packages-template/bbb-html5/
-78,bbb-common-web/src/main/java/org/bigbluebutton/presentation/
-79,bbb-common-web/src/main/java/org/bigbluebutton/presentation/
+66,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
+66,bbb-fsesl-client/src/main/java/org/freeswitch/
+66,bbb-fsesl-client/src/test/java/org/freeswitch/
+66,build/packages-template/bbb-freeswitch-core/
+66,build/packages-template/bbb-freeswitch-sounds/
+66,freeswitch.placeholder.sh
+67,bbb-webrtc-sfu.placeholder.sh
+67,build/packages-template/bbb-webrtc-sfu/
+70,bbb-webrtc-sfu.placeholder.sh
+70,build/packages-template/bbb-webrtc-sfu/
+72,akka-bbb-fsesl/src/main/java/org/bigbluebutton/freeswitch/
+72,bbb-fsesl-client/src/main/java/org/freeswitch/
+72,bbb-fsesl-client/src/test/java/org/freeswitch/
+72,bigbluebutton-html5/
+72,build/packages-template/bbb-freeswitch-core/
+72,build/packages-template/bbb-freeswitch-sounds/
+72,build/packages-template/bbb-html5/
+72,freeswitch.placeholder.sh
+73,bbb-webrtc-sfu.placeholder.sh
+73,bigbluebutton-html5/
+73,build/packages-template/bbb-html5/
+73,build/packages-template/bbb-webrtc-sfu/
+76,bigbluebutton-html5/
+76,build/packages-template/bbb-html5/
+78,bbb-common-web/src/main/java/org/bigbluebutton/web/
+78,bigbluebutton-web/
+78,build/packages-template/bbb-web/
+79,bbb-common-message/src/main/java/org/bigbluebutton/common2/redis/pubsub/
+79,bigbluebutton-html5/
+79,build/packages-template/bbb-html5/
+80,bbb-common-web/src/main/java/org/bigbluebutton/presentation/
+81,bbb-common-web/src/main/java/org/bigbluebutton/presentation/

bigbluebutton/goldstandards/goldstandard_sad_2021-sam_2021.csv

@@ -19,46 +19,45 @@ _yGgUMFkHEeyewPSmlgszyA,19
 _0e5u8FkHEeyewPSmlgszyA,19
 _yGgUMFkHEeyewPSmlgszyA,20
 _yGgUMFkHEeyewPSmlgszyA,21
-_XuNf4FkJEeyewPSmlgszyA,25
-_4yx-8FkHEeyewPSmlgszyA,28
-_4yx-8FkHEeyewPSmlgszyA,34
-_4yx-8FkHEeyewPSmlgszyA,35
+_XuNf4FkJEeyewPSmlgszyA,26
+_4yx-8FkHEeyewPSmlgszyA,30
 _4yx-8FkHEeyewPSmlgszyA,36
-_yGgUMFkHEeyewPSmlgszyA,37
-_6sergFkHEeyewPSmlgszyA,44
-_6sergFkHEeyewPSmlgszyA,45
-_yGgUMFkHEeyewPSmlgszyA,45
-_RDHFIFwPEeyiuNx_RO7j-Q,46
-_RDHFIFwPEeyiuNx_RO7j-Q,47
-_XuNf4FkJEeyewPSmlgszyA,49
-_XuNf4FkJEeyewPSmlgszyA,50
+_4yx-8FkHEeyewPSmlgszyA,37
+_4yx-8FkHEeyewPSmlgszyA,38
+_yGgUMFkHEeyewPSmlgszyA,39
+_6sergFkHEeyewPSmlgszyA,46
+_6sergFkHEeyewPSmlgszyA,47
+_yGgUMFkHEeyewPSmlgszyA,47
+_RDHFIFwPEeyiuNx_RO7j-Q,48
+_RDHFIFwPEeyiuNx_RO7j-Q,49
 _XuNf4FkJEeyewPSmlgszyA,51
 _XuNf4FkJEeyewPSmlgszyA,52
-_9wZIcFkHEeyewPSmlgszyA,55
-_nwrCMFwPEeyiuNx_RO7j-Q,55
-_nwrCMFwPEeyiuNx_RO7j-Q,56
-_9wZIcFkHEeyewPSmlgszyA,56
-_9wZIcFkHEeyewPSmlgszyA,58
-_XuNf4FkJEeyewPSmlgszyA,58
-_6sergFkHEeyewPSmlgszyA,58
+_XuNf4FkJEeyewPSmlgszyA,53
+_XuNf4FkJEeyewPSmlgszyA,54
+_9wZIcFkHEeyewPSmlgszyA,57
+_nwrCMFwPEeyiuNx_RO7j-Q,58
 _nwrCMFwPEeyiuNx_RO7j-Q,59
-_nwrCMFwPEeyiuNx_RO7j-Q,60
+_9wZIcFkHEeyewPSmlgszyA,60
+_XuNf4FkJEeyewPSmlgszyA,60
+_6sergFkHEeyewPSmlgszyA,60
 _nwrCMFwPEeyiuNx_RO7j-Q,61
-_s0aIcFkHEeyewPSmlgszyA,63
-_nwrCMFwPEeyiuNx_RO7j-Q,64
+_nwrCMFwPEeyiuNx_RO7j-Q,62
+_nwrCMFwPEeyiuNx_RO7j-Q,63
 _s0aIcFkHEeyewPSmlgszyA,65
-_oN4CMFkHEeyewPSmlgszyA,65
-_oN4CMFkHEeyewPSmlgszyA,66
+_nwrCMFwPEeyiuNx_RO7j-Q,66
+_s0aIcFkHEeyewPSmlgszyA,67
 _oN4CMFkHEeyewPSmlgszyA,67
-_s0aIcFkHEeyewPSmlgszyA,68
-_0e5u8FkHEeyewPSmlgszyA,70
-_nwrCMFwPEeyiuNx_RO7j-Q,70
-_0e5u8FkHEeyewPSmlgszyA,71
-_yGgUMFkHEeyewPSmlgszyA,71
-_s0aIcFkHEeyewPSmlgszyA,71
-_0e5u8FkHEeyewPSmlgszyA,74
-_4yx-8FkHEeyewPSmlgszyA,76
-_6sergFkHEeyewPSmlgszyA,77
-_0e5u8FkHEeyewPSmlgszyA,77
-_2BgxMFkHEeyewPSmlgszyA,78
-_2BgxMFkHEeyewPSmlgszyA,79
+_oN4CMFkHEeyewPSmlgszyA,68
+_oN4CMFkHEeyewPSmlgszyA,69
+_s0aIcFkHEeyewPSmlgszyA,70
+_0e5u8FkHEeyewPSmlgszyA,72
+_nwrCMFwPEeyiuNx_RO7j-Q,72
+_0e5u8FkHEeyewPSmlgszyA,73
+_yGgUMFkHEeyewPSmlgszyA,73
+_s0aIcFkHEeyewPSmlgszyA,73
+_0e5u8FkHEeyewPSmlgszyA,76
+_4yx-8FkHEeyewPSmlgszyA,78
+_6sergFkHEeyewPSmlgszyA,79
+_0e5u8FkHEeyewPSmlgszyA,79
+_2BgxMFkHEeyewPSmlgszyA,80
+_2BgxMFkHEeyewPSmlgszyA,81