Reproducible Output

Deterministic Generation by Default

All generation functions in localLLM (quick_llama(), generate(), and generate_parallel()) use deterministic greedy decoding by default. This means running the same prompt twice will produce identical results.

library(localLLM)

# Run the same query twice
response1 <- quick_llama("What is the capital of France?")
response2 <- quick_llama("What is the capital of France?")

# Results are identical
identical(response1, response2)

#> [1] TRUE

Seed Control for Stochastic Generation

Reproducibility is ensured even when temperature > 0:

# Stochastic generation with seed control
response1 <- quick_llama(
  "Write a haiku about data science",
  temperature = 0.9,
  seed = 92092
)

response2 <- quick_llama(
  "Write a haiku about data science",
  temperature = 0.9,
  seed = 92092
)

# Still reproducible with matching seeds
identical(response1, response2)

#> [1] TRUE

# Different seeds produce different outputs
response3 <- quick_llama(
  "Write a haiku about data science",
  temperature = 0.9,
  seed = 12345
)

identical(response1, response3)

#> [1] FALSE

Input/Output Hash Verification

All generation functions compute SHA-256 hashes for both inputs and outputs. These hashes enable verification that collaborators used identical configurations and obtained matching results.

result <- quick_llama("What is machine learning?")

# Access the hashes
hashes <- attr(result, "hashes")
print(hashes)

#> $input
#> [1] "a3f2b8c9d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1"
#>
#> $output
#> [1] "b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5"

The input hash includes: - Model identifier - Prompt text - Generation parameters (temperature, seed, max_tokens, etc.)

The output hash covers the generated text, allowing collaborators to verify they obtained matching results.

res <- explore(
  models = models,
  prompts = template_builder,
  hash = TRUE
)

# View hashes for each model
hash_df <- attr(res, "hashes")
print(hash_df)

#>   model_id                         input_hash                        output_hash
#> 1  gemma4b a3f2b8c9d4e5f6a7b8c9d0e1f2a3b4c5... b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9...
#> 2  llama3b c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0... d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1...

Automatic Documentation

Use document_start() and document_end() to capture everything that happens during your analysis. The log records:

• Timestamps
• Model metadata (paths, parameters)
• Summaries of function calls
• SHA-256 fingerprint of the entire run

# Start documentation
document_start(path = "analysis-log.txt")

# Run your analysis
result1 <- quick_llama("Classify this text: 'Great product!'")
result2 <- explore(models = models, prompts = prompts)

# End documentation
document_end()

The log file contains a complete audit trail:

localLLM Run Log
File: /path/to/analysis-log.txt
Started: 2025-01-15 14:30:22 EST
Ended: 2025-01-15 14:35:12 EST
Duration: 289.9 seconds

Events:
- [2025-01-15 14:30:22 EST] document_start
    {
      "package_version": "1.4.1",
      "r_version": "4.4.1",
      "platform": "aarch64-apple-darwin22.6.0",
      "os": "Darwin",
      "user": "researcher",
      "working_directory": "/home/user/analysis"
    }

- [2025-01-15 14:30:25 EST] quick_llama
    {
      "model": "Llama-3.2-3B-Instruct-Q5_K_M.gguf",
      "prompt_count": 1,
      "n_gpu_layers": 999,
      "n_ctx": 2048,
      "max_tokens": 100,
      "temperature": 0,
      "seed": 1234,
      "auto_format": true,
      "clean": false
    }

- [2025-01-15 14:30:25 EST] quick_llama_hash
    {
      "input_hash": "a3f2b8c9...",
      "output_hash": "b4c5d6e7..."
    }

- [2025-01-15 14:35:12 EST] document_end
    {
      "duration_seconds": 289.9,
      "total_events": 4
    }

Hash (SHA-256): e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2...

Best Practices for Reproducible Research

result <- quick_llama(
  "Analyze this text",
  temperature = 0,
  seed = 42  # Explicit for documentation
)

# Check hardware profile
hw <- hardware_profile()
print(hw)

#> $os
#> [1] "Darwin"
#>
#> $cpu_cores
#> [1] 10
#>
#> $ram_total
#> [1] 17179869184
#>
#> $gpu
#> $gpu$name
#> [1] "Apple M2 Pro"

document_start(path = "my_analysis_log.txt")

# All your analysis code here
# ...

document_end()

result <- quick_llama("Your prompt here", seed = 42)

# Report these in your paper/documentation
cat("Input hash:", attr(result, "hashes")$input, "\n")
cat("Output hash:", attr(result, "hashes")$output, "\n")

# List cached models with metadata
cached <- list_cached_models()
print(cached[, c("name", "size_bytes", "modified")])

packageVersion("localLLM")

# install.packages("remotes")

# From the CRAN archive, e.g. version 1.3.0
remotes::install_version("localLLM", version = "1.3.0")

# Or from the matching GitHub release tag
remotes::install_github("EddieYang211/localLLM@v1.3.0", subdir = "localLLM")

library(localLLM)
install_localLLM()          # downloads the backend for this package version
packageVersion("localLLM")  # confirm the version

What Reproducibility Covers

Repeated runs give identical output when all of the following are the same:

• the model file,
• the localLLM package and backend version,
• the hardware, the GPU backend, and the number of threads,
• the generation parameters and the seed.

A few caveats:

• CPU: llama.cpp splits work across threads in a fixed way, so identical output on the same machine with the same n_threads is expected.
• GPU (Metal, Vulkan): repeated runs on the same machine were identical in our tests, but bit-for-bit identity is not a formal guarantee of the GPU libraries.
• Different machines: results can differ across hardware. For example, llama.cpp tunes some Metal kernels per Apple chip, and CPU builds use different instruction sets.
• Mixture-of-experts (MoE) models have an extra source of variation in expert routing.

When exact replication matters, share the model file (or its hash), the package version, and your hardware profile together with the output hashes.

Summary

With these tools, your LLM-based analyses can be reproduced and verified on the same software and hardware.