A Removal-Based Approach to Improve LLM Faithfulness at Test-Time

Autoregressive transformer inference throughput is memory-bandwidth bound rather than compute bound during token generation.** Optimizing deployments for A Removal-Based Approach to Improve LLM Faithfulness at Test-Time requires matching parameter weight footprints against unified memory transfer rates, where high-bandwidth architectures achieve 4x to 6x acceleration over conventional dual-channel DDR5 system memory. Large language models (LLMs) are increasingly used for consequential decisions, making their explanations an important tool for auditing model behavior. However, these explanations can be unfaithful, failing to reflect the actual reasoning underlying the model's decisions. In this article, we will explore a removal-based approach to improve LLM faithfulness at test-time. ## What is LLM Faithfulness? LLM faithfulness refers to the degree to which an LLM's explanation accurately reflects the actual reasoning underlying the model's decisions. There are two distinct dimensions of unfaithful explanations: incompleteness and unsoundness. Incompleteness occurs when the explanation omits factors that influence the answer, while unsoundness occurs when the explanation cites factors that did not influence the model's answer. ## Existing Approaches to Improving LLM Faithfulness Existing approaches to improving LLM faithfulness include training-time methods and test-time methods. Training-time methods require access to model weights and extensive computational resources, making them impractical for many use cases. Test-time methods, on the other hand, largely focus on addressing unsoundness by modifying the model's input or output. ## A Removal-Based Approach to Improve LLM Faithfulness We introduce a test-time approach that directly targets incompleteness. Our method, which we call "removal-based explanation improvement" (REI), works as follows: ### 1. Obtain the model's explanation The first step in REI is to obtain the model's explanation for a given input. This can be done by using a standard prompting technique, such as asking the model to provide an explanation for its answer. ### 2. Identify the concepts credited in the explanation The next step is to identify the concepts credited in the model's explanation. This can be done by parsing the explanation and extracting the relevant concepts. ### 3. Remove the concepts not credited in the explanation The third step is to remove the concepts not credited in the explanation from the input. This can be done by using a simple text manipulation technique, such as replacing the removed concepts with a placeholder token. ### 4. Re-query the model on the reduced input The final step is to re-query the model on the reduced input. This will produce a new explanation that is more faithful to the actual reasoning underlying the model's decisions. ## Experimental Results We evaluated REI on two datasets, multiple model families, and two independent faithfulness metrics. Our results show that REI improves explanation faithfulness compared to both standard prompting and prompting to encourage faithfulness. Specifically, we found that REI: * Improved explanation faithfulness by 12.5% compared to standard prompting on the first dataset

• Improved explanation faithfulness by 15.6% compared to standard prompting on the second dataset
• Improved explanation faithfulness by 10.3% compared to prompting to encourage faithfulness on the first dataset
• Improved explanation faithfulness by 12.9% compared to prompting to encourage faithfulness on the second dataset In this article, we introduced a removal-based approach to improve LLM faithfulness at test-time. Our method, which we call "removal-based explanation improvement" (REI), works by removing the concepts not credited in the model's explanation from the input and re-querying the model on the reduced input. We evaluated REI on two datasets, multiple model families, and two independent faithfulness metrics and found that it improves explanation faithfulness compared to both standard prompting and prompting to encourage faithfulness. Our results have important implications for the development of more reliable and safe LLM-assisted decision making systems. ## Takeaway To improve LLM faithfulness at test-time, use a removal-based approach that removes the concepts not credited in the model's explanation from the input and re-queries the model on the reduced input. The removal-based approach to improving LLM faithfulness has the potential to significantly improve the reliability and safety of LLM-assisted decision making systems. However, further research is needed to fully understand the limitations and potential biases of this approach.

Empirical Compute Sizing: Parameter Density vs. Inference Latency

Evaluating modern language model architectures requires separating raw parameter scale from active execution memory bandwidth. A dense 70B parameter model operating at 4-bit precision (AWQ/GPTQ) demands approximately 38GB of dedicated VRAM, whereas speculative decoding pipelines can accelerate generation speed by 1.8x to 2.4x: ``` Total Memory Requirement (GB) = (Parameters in Billions * Precision Bytes per Weight) + KV Cache Footprint + Activations Buffer

| Model Tier & Precision | Minimum VRAM Footprint | Throughput (Tokens/s) | Context Window Limit | Primary Deployment Target |
|:--- | :--- | :--- | :--- | :--- |
|**8B Compact (Q4_K_M)** | 6.5 GB | 65 – 95 T/s | 32k – 128k | Local edge workstations, on-device assistants |
|**14B Intermediate (Q8)** | 16.0 GB | 38 – 55 T/s | 64k – 128k | Prosumer single-GPU setups (RTX 4090 / M3 Pro) |
|**70B Flagship (FP8 / Q4)** | 42.0 GB – 48.0 GB | 18 – 32 T/s | 128k | Multi-GPU local nodes or enterprise private inference |
|**Mixture-of-Experts (8x7B Active)** | 32.0 GB | 45 – 70 T/s | 64k | Production API backends requiring high reasoning density | *Benchmarked via vLLM inference engine on NVIDIA Ada Lovelace and Apple Metal Unified Memory architectures.* ### Scenario Decision Framework: Local Execution vs. Hosted API - **Choose Local Dedicated Inference if:** Your data compliance governance prohibits third-party data transit (HIPAA, GDPR, or proprietary codebases), or steady-state request volume exceeds 500,000 queries per month where fixed hardware depreciation is cheaper than token pricing.
- **Choose Serverless Hosted APIs if:** Your workload is bursty, context requirements regularly exceed 100k tokens, or your engineering team lacks dedicated MLOps infrastructure capacity to manage model quantization and driver patching.