A leakage-audited comparison of neural architectures for fake news detection: a cross-dataset evaluation

Mohammad Alkhazaleh, Mahmoud Baklizi, Musab Alzghoul, Loui Al Sardy

Abstract


Reported accuracies for fake news detection span roughly 60% to 99%, too wide a spread to be explained by architecture. We trace much of it to leakage. Auditing a corpus built from word embedding over linguistic features for fake (WELFake), LIAR, FakeNewsNet, and a published COVID-19 dataset, we find five publisher artifacts, chiefly wire-service datelines and image-credit lines, in 31,701 of the 96,626 documents collected (32.8%); they predict the label at about 99.7% purity by string matching alone, and fake articles are exact duplicates fifteen times more often than real ones. They survive preprocessing and are not removed by source-aware splitting, because they sit inside the article. After mitigating both channels and partitioning 87,408 deduplicated articles by source, we compare seven architectures under three seeds each and detect no robust separation: the spread between their macro-averaged accuracies (2.16 points) is smaller than the 2.63 expected from seed variation alone, an absence of detected separation rather than evidence of equivalence. A term frequency-inverse document frequency (TF-IDF) logistic regression (LR) fitted in 77 seconds reaches 88.05% against 89.30% for the best neural model. WELFake supplies 69.6 of those 89.3 points, and on LIAR every run gives an area under the curve (AUC) of 0.57-0.59. Predictions, splits and an audit tool are released.

Keywords


Cross-dataset evaluation; Data leakage; Fake news detection; Negative results; Reproducibility; Text classification

Full Text:

PDF


DOI: https://doi.org/10.11591/eei.v15i6.12782

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Bulletin of EEI Statistics

Bulletin of Electrical Engineering and Informatics (BEEI)
ISSN: 2089-3191 , e-ISSN: 2302-9285
This journal is published by the Institute of Advanced Engineering and Science (IAES) .