specoto/besian-ai-stats

Bayesian Statistics Library for AI Code Generation Predictability Metrics

dev-main 2026-02-09 11:45 UTC

Requires

• php: >=8.0
• ext-json: *
• ext-mbstring: *

Requires (Dev)

• phpstan/phpstan: ^1.0
• phpunit/phpunit: ^9.0

GPL-2.0-or-later

• Maintainer

• Metrics
• statistics
• ai
• code-generation
• bayesian
• predictability

README

A comprehensive PHP library for analyzing AI-generated code quality using Bayesian statistical methods. This library provides probabilistic analysis of code quality, maintainability, and predictability with interactive visualization capabilities.

🚀 Features

• Bayesian Analysis: Posterior distributions for code quality metrics
• AI Code Analysis: Enhanced analysis with prompt and generation context
• Code Sampling: Git history analysis and directory scanning
• Risk Assessment: Probabilistic risk evaluation with confidence intervals
• Interactive Charts: Multiple visualization types (bar, radar, doughnut, line)
• Real-time Monitoring: Live dashboard with automatic updates
• Configuration Management: Flexible JSON-based configuration system
• Context-Aware Metrics: Reliability assessment based on developer skill and intent

📦 Installation

Composer Installation

composer require specoto/besian-ai-stats

Manual Installation

1. Clone or download the library
2. Include the autoloader in your project:

require_once 'vendor/autoload.php';

🔧 Configuration

Quick Setup

1. Copy Example Configuration

cp vendor/specoto/besian-ai-stats/config.example.json besian_ai_stats_config.json

1. Create Data Directory

mkdir -p data

1. Configure for Your Project

Edit besian_ai_stats_config.json to match your project structure:

{
    "code_sampling": {
        "git_history": {
            "file_patterns": ["src/**/*.php", "lib/**/*.php"],
            "max_commits": 100
        }
    },
    "bayesian_analysis": {
        "persistence_file": "data/besian_posteriors.json",
        "training_data_export_file": "data/git_training_analysis.json"
    }
}

Configuration Options

Code Sampling

• git_history: Analyze git commit history
• current_directory: Scan current project directory
• max_commits: Number of commits to analyze (default: 100)
• file_patterns: Patterns for eligible files
• exclude_patterns: Patterns to exclude from analysis

Bayesian Analysis

• persistence_file: Location for posterior distributions storage
• training_data_export_file: Location for training results
• confidence_threshold: Minimum confidence for quality metrics
• learning_rate: Bayesian learning rate (default: 0.1)

Quality Thresholds

• min_lines_of_code: Minimum lines for analysis (default: 10)
• max_complexity: Maximum allowed complexity (default: 10)
• require_documentation: Require documentation (default: true)
• min_test_coverage: Minimum test coverage percentage (default: 0.6)

Prompt and Context Sources

The library supports configurable sources for extracting prompts and generation context:

{
    "prompt_context_sources": {
        "enabled": true,
        "prompt_sources": {
            "git_commit_message": {
                "enabled": true,
                "priority": 1,
                "max_length": 200,
                "cleanup_patterns": [
                    "^(fix|feat|docs|style|refactor|test|chore)(\\([^)]+\\))?:\\s*"
                ]
            },
            "file_path_analysis": {
                "enabled": true,
                "priority": 2,
                "path_mappings": {
                    "src/Controller": "Generate controller for {feature}",
                    "src/Service": "Implement service for {feature}",
                    "tests/": "Write tests for {feature}"
                }
            }
        },
        "context_sources": {
            "git_author_analysis": {
                "enabled": true,
                "priority": 1,
                "author_experience_mapping": {
                    "default": 5,
                    "senior_developers": ["john.doe", "jane.smith"],
                    "junior_developers": ["intern.user", "newbie.dev"]
                },
                "experience_levels": {
                    "senior": 7,
                    "junior": 3
                }
            },
            "domain_knowledge_inference": {
                "enabled": true,
                "priority": 2,
                "domain_mappings": {
                    "src/Controller": {"web_development": 7, "http": 6},
                    "src/Service": {"business_logic": 8, "architecture": 6},
                    "tests/": {"testing": 8, "quality_assurance": 7}
                }
            }
        },
        "fallback_strategies": {
            "prompt_fallbacks": [
                "Generate {file_type} code for {filename}",
                "Analyze {filename} implementation"
            ],
            "context_fallbacks": {
                "developer_experience": 5,
                "domain_knowledge": 5,
                "complexity_level": "medium"
            }
        }
    }
}

Available Prompt Sources:

• git_commit_message: Extract from git commit messages with cleanup patterns
• git_diff_analysis: Analyze git diff changes to infer intent
• file_path_analysis: Generate prompts based on file structure
• ai_generation_logs: Read from AI generation log files
• custom_metadata: Extract from custom metadata files

Available Context Sources:

• git_author_analysis: Infer developer experience from author patterns
• domain_knowledge_inference: Determine domain expertise from file paths
• tool_detection: Detect development tools used
• complexity_analysis: Analyze code complexity metrics
• team_context_analysis: Infer team size and collaboration patterns

📊 Real Code Quality Tool Integration

Supported PHP Tools

Code Quality Analysis:

• PHPStan: Static analysis with customizable levels (0-9)
• PHPCS: Coding standards compliance (PSR-12, Symfony, etc.)
• PHPMD: Mess detection with customizable rule sets
• PHPUnit: Test coverage and test result analysis

Security Analysis:

• Psalm: Type checking and security vulnerability detection
• Phan: Static analysis with security focus
• Security Checker: Composer dependency vulnerability scanning
• Composer Audit: Built-in dependency vulnerability analysis

Custom Tools:

• ESLint: JavaScript/TypeScript linting
• Stylelint: CSS/SCSS style checking
• Any CLI tool: Custom command integration with flexible output parsing

Tool Configuration

{
    "code_quality_tools": {
        "enabled": true,
        "project_root": ".",
        "php_tools": {
            "phpstan": {
                "enabled": true,
                "priority": 1,
                "binary_path": "vendor/bin/phpstan",
                "level": 5,
                "config_file": "phpstan.neon",
                "memory_limit": "1G",
                "metrics": ["syntax_correctness", "logical_correctness", "maintainability"]
            },
            "phpcs": {
                "enabled": true,
                "priority": 2,
                "standard": ["PSR-12"],
                "metrics": ["syntax_correctness", "maintainability", "documentation_quality"]
            }
        },
        "security_tools": {
            "psalm": {
                "enabled": true,
                "level": 3,
                "config_file": "psalm.xml",
                "metrics": ["security_compliance", "syntax_correctness"]
            },
            "composer_audit": {
                "enabled": true,
                "metrics": ["security_compliance", "scalability"]
            }
        }
    }
}

Security Analysis Example

<?php

require_once 'vendor/autoload.php';

use BesanAIStats\AICodeMetricsAnalyzer;

// Security-focused configuration
$securityConfig = [
    'code_quality_tools' => [
        'php_tools' => [
            'phpstan' => ['enabled' => true, 'level' => 8],
            'phpcs' => ['enabled' => true]
        ],
        'security_tools' => [
            'psalm' => ['enabled' => true, 'level' => 5],
            'composer_audit' => ['enabled' => true],
            'security_checker' => ['enabled' => true]
        ]
    ]
];

$analyzer = new AICodeMetricsAnalyzer([], [], null, $securityConfig);

$codeMetrics = [
    'raw_code' => '<?php
class AuthController {
    public function login(string $username, string $password): bool {
        $query = "SELECT * FROM users WHERE username = \'' . $username . '\' AND password = \'' . $password . '\'";
        // SQL injection vulnerability!
        return true;
    }
}',
    'prompt' => 'Create authentication controller'
];

$result = $analyzer->analyzeCodeQuality($codeMetrics);
$context = $result->getContext();

echo "Security Compliance: " . ($context['security_compliance'] * 100) . "%\n";
echo "Security Issues: " . ($context['security_vulnerabilities_count'] ?? 0) . "\n";

📊 Usage Examples

Basic Usage

<?php

require_once 'vendor/autoload.php';

use BesanAIStats\BayesianAIStats;

// Initialize with custom storage location
$bayesianStats = new BayesianAIStats(
    [], // Initial data
    '/custom/path/to/data/besian_posteriors.json' // Custom storage
);

// Analyze code quality
$metrics = [
    'syntax_correctness' => 0.9,
    'logical_correctness' => 0.8,
    'performance_optimization' => 0.6,
    'security_compliance' => 0.85,
    'maintainability' => 0.7,
    'test_coverage' => 0.4,
    'documentation_quality' => 0.3,
    'scalability' => 0.65,
];

$result = $bayesianStats->calculatePredictabilityScore($metrics);

echo "Overall Quality Score: " . ($result->getOverallScore() * 100) . "%\n";
echo "Confidence: " . ($result->getOverallConfidence() * 100) . "%\n";

// Save results
$bayesianStats->savePosteriors();

AI Code Analysis with Prompt and Context

<?php

require_once 'vendor/autoload.php';

use BesanAIStats\AICodeMetricsAnalyzer;

// Initialize the AI code metrics analyzer
$analyzer = new AICodeMetricsAnalyzer();

// Analyze AI-generated code with prompt and generation context
$codeMetrics = [
    'raw_code' => '<?php echo "Hello, World!"; ?>',
    'prompt' => 'Generate a simple PHP echo statement that outputs "Hello, World!"',
    'generation_context' => [
        'developer_experience' => 7,
        'domain_knowledge' => 6,
        'complexity_level' => 'low',
        'ai_model_used' => 'GPT-4',
        'temperature' => 0.7,
        'tools_used' => ['VSCode', 'Git'],
        'time_constraint' => 'standard'
    ]
];

// Analyze the code with additional context
$additionalContext = [
    'project_name' => 'Web Application',
    'framework' => 'Drupal',
    'team_size' => 5
];

$result = $analyzer->analyzeCodeQuality($codeMetrics, $additionalContext);

// Get comprehensive analysis results
$predictability = $result->getPredictabilityResult();
$context = $result->getContext();

echo "Quality Score: " . ($predictability->getOverallScore() * 100) . "%\n";
echo "Confidence: " . ($predictability->getOverallConfidence() * 100) . "%\n";
echo "Production Ready: " . ($result->isProductionReady() ? 'Yes' : 'No') . "\n";

// Access prompt and generation context used for analysis
echo "Original Prompt: " . ($context['prompt'] ?? 'Not provided') . "\n";
echo "Developer Experience: " . ($context['generation_context']['developer_experience'] ?? 'N/A') . "/10\n";
echo "Complexity Level: " . ($context['generation_context']['complexity_level'] ?? 'N/A') . "\n";

Git-Based Training with Prompt Context

<?php

require_once 'vendor/autoload.php';

use BesanAIStats\GitCodeSampler;
use BesanAIStats\AICodeMetricsAnalyzer;

// Initialize the code sampler
$sampler = new GitCodeSampler($config);

// Generate training data from git history
$trainingData = $sampler->generateTrainingData();

foreach ($trainingData as $sample) {
    $analyzer = new AICodeMetricsAnalyzer($sample);
    
    // Prepare code metrics with prompt and generation context
    $codeMetrics = [
        'raw_code' => $sample['content'],
        'prompt' => $sample['commit_message'] ?? "Analyze {$sample['file']}",
        'generation_context' => [
            'developer_experience' => 5, // Default medium level
            'domain_knowledge' => 5,
            'complexity_level' => 'medium',
            'file_type' => pathinfo($sample['file'], PATHINFO_EXTENSION),
            'git_author' => $sample['author'] ?? 'unknown',
            'commit_hash' => $sample['commit_hash'] ?? 'current'
        ]
    ];
    
    $result = $analyzer->analyzeCodeQuality($codeMetrics, $sample);
    
    // The result now contains prompt and context in its context
    $context = $result->getContext();
    echo "File: {$sample['file']}\n";
    echo "Prompt: " . substr($context['prompt'], 0, 50) . "...\n";
    echo "Developer Experience: " . $context['generation_context']['developer_experience'] . "/10\n";
    echo "Quality Score: " . ($result->getPredictabilityResult()->getOverallScore() * 100) . "%\n\n";
}

Advanced Usage with Training Data

<?php

require_once 'vendor/autoload.php';

use BesanAIStats\BayesianAIStats;

// Load configuration
$configFile = 'besian_ai_stats_config.json';
$config = [];
if (file_exists($configFile)) {
    $config = json_decode(file_get_contents($configFile), true);
}

// Initialize with configuration
$bayesianStats = new BayesianAIStats([], $config['bayesian_analysis']['persistence_file'] ?? null);

// Analyze multiple code samples
$trainingData = [
    [
        'file' => 'src/Controller/UserController.php',
        'syntax_correctness' => 0.95,
        'logical_correctness' => 0.85,
        'performance_optimization' => 0.7,
        'security_compliance' => 0.9,
        'confidence' => 0.8,
    ],
    [
        'file' => 'src/Service/AuthService.php',
        'syntax_correctness' => 0.8,
        'logical_correctness' => 0.75,
        'performance_optimization' => 0.6,
        'security_compliance' => 0.85,
        'confidence' => 0.7,
    ],
];

foreach ($trainingData as $dataPoint) {
    $bayesianStats->addTrainingData($dataPoint);
}

// Get updated posterior distributions
$posteriors = $bayesianStats->getPosteriors();
foreach ($posteriors as $metric => $distribution) {
    echo "$metric: " . ($distribution->getMean() * 100) . "% (confidence: " . ($distribution->getVariance() < 0.01 ? 'high' : 'medium') . ")\n";
}

📈 Data Analysis

Supported Metrics

Metric	Description	Range	Interpretation
Syntax Correctness	Code syntax validation	0-100%	Higher is better
Logical Correctness	Code logic assessment	0-100%	Higher is better
Performance Optimization	Efficiency analysis	0-100%	Higher is better
Security Compliance	Security standards	0-100%	Higher is better
Maintainability	Code maintenance	0-100%	Higher is better
Test Coverage	Unit test coverage	0-100%	Higher is better
Documentation Quality	Code documentation	0-100%	Higher is better
Scalability	System scalability	0-100%	Higher is better

AI Context Integration

The library now supports prompt and generation context to enhance reliability assessment:

Prompt Context

• Purpose: Captures the original intent behind AI-generated code
• Source: User prompts, commit messages, or task descriptions
• Impact: Helps AI understand expected behavior and requirements

Generation Context

• Developer Experience (1-10): Skill level of the developer
• Domain Knowledge (1-10): Familiarity with the problem domain
• Complexity Level: low, medium, or high task complexity
• AI Model Used: Which AI model generated the code
• Temperature: Creativity/randomness setting used
• Tools Used: Development tools and IDEs involved
• Time Constraint: Development time pressure
• Team Size: Number of developers involved

Benefits

1. Improved Accuracy: Context helps AI better assess code quality relative to intent
2. Skill-Based Assessment: Accounts for developer experience levels
3. Intent Understanding: Prompts clarify what the code was supposed to achieve
4. Better Risk Assessment: Considers complexity and time constraints
5. Enhanced Training Data: Provides richer information for model improvement

Usage Example

$codeMetrics = [
    'raw_code' => '<?php echo "Hello"; ?>',
    'prompt' => 'Create a greeting message',
    'generation_context' => [
        'developer_experience' => 8,
        'domain_knowledge' => 7,
        'complexity_level' => 'low'
    ]
];

$result = $analyzer->analyzeCodeQuality($codeMetrics);
$context = $result->getContext();

// Access context data
$prompt = $context['prompt'];
$devExp = $context['generation_context']['developer_experience'];

Risk Assessment

The library provides probabilistic risk assessment based on Bayesian analysis:

• Low Risk: Quality score ≥ 70%, confidence ≥ 80%
• Medium Risk: Quality score 50-70%, confidence 60-80%
• High Risk: Quality score < 50%, confidence < 60%

📊 Bayesian Posteriors Explained

What Are Bayesian Posteriors?

In Bayesian statistics, a posterior distribution represents our updated beliefs about a parameter after observing evidence. Unlike fixed parameters, posteriors are probability distributions that evolve as we gather more data.

Key Concepts

Prior Distribution: Initial belief before seeing data Likelihood: Probability of observing data given a parameter value Posterior Distribution: Updated belief after observing data (Prior × Likelihood)

Beta Distribution: Most common posterior for binary outcomes (success/failure, good/bad)

Posterior Components

For each quality metric, the library tracks:

• Alpha (α): Number of successful observations + prior successes
• Beta (β): Number of failed observations + prior failures
• Mean: Expected value = α / (α + β)
• Variance: Uncertainty = (α × β) / ((α + β)² × (α + β + 1))

Our Metrics Explained

Metric	What It Measures	Prior	How It Updates	What It Means
Syntax Correctness	Code follows language syntax	Beta(2,1) - Optimistic	More valid syntax → Higher α	Code likely has correct syntax
Logical Correctness	Code logic is sound	Beta(1.5,1.5) - Neutral	Balanced prior	Good logic → Higher α	Code has logical consistency
Performance	Code runs efficiently	Beta(1,2) - Conservative	Low initial belief	Performance evidence → Higher α	Code is optimized
Security	Code follows security practices	Beta(3,1) - Optimistic	Security-first prior	Security practices → Higher α	Code is secure
Maintainability	Code is easy to modify	Beta(1,2) - Conservative	Maintainable code is rare	Good practices → Higher α	Code is maintainable
Test Coverage	Code has tests	Beta(1.5,1.5) - Neutral	Balanced testing	Test evidence → Higher α	Code is well-tested
Documentation	Code is documented	Beta(1,3) - Pessimistic	Documentation is often poor	Good docs → Higher α	Code is documented
Scalability	Code scales well	Beta(1,2) - Conservative	Scalable code is rare	Good design → Higher α	Code scales properly

Practical Example

// Initial posterior for syntax correctness: Beta(α=2, β=1)
// Mean belief: 2/(2+1) = 66.7% code has correct syntax

// After analyzing 10 files with 8 correct syntax, 2 errors:
// New posterior: Beta(α=2+8=10, β=1+2=3) 
// New mean: 10/(10+3) = 76.9% code has correct syntax

// Higher confidence because we have more evidence (larger α+β)

Why This Matters

1. Uncertainty Quantification: Posterior distributions tell us how certain we are about each metric
2. Adaptive Learning: The system becomes more accurate with more data
3. Risk Assessment: Mean and variance help identify high-risk areas
4. Probabilistic Predictions: Future quality can be predicted with confidence intervals

Data Visualization

Posterior distributions can be visualized as:

• Probability Density Curves: Show full uncertainty landscape
• Credibility Intervals: 95% confidence ranges for predictions
• Update Over Time: Watch how beliefs evolve with more evidence
• Risk Heat Maps: Combine multiple metrics for overall risk assessment

Supported Metrics

The library analyzes these code quality metrics using Bayesian posterior distributions:

Metric	Description	Range	Interpretation
Syntax Correctness	Code syntax validation	0-100%	Higher is better
Logical Correctness	Code logic assessment	0-100%	Higher is better
Performance Optimization	Efficiency analysis	0-100%	Higher is better
Security Compliance	Security standards	0-100%	Higher is better
Maintainability	Code maintenance	0-100%	Higher is better
Test Coverage	Unit test coverage	0-100%	Higher is better
Documentation Quality	Code documentation	0-100%	Higher is better
Scalability	System scalability	0-100%	Higher is better

Risk Assessment

The library provides probabilistic risk assessment based on Bayesian analysis:

• Low Risk: Quality score ≥ 70%, confidence ≥ 80%
• Medium Risk: Quality score 50-70%, confidence 60-80%
• High Risk: Quality score < 50%, confidence < 60%

📊 Visualization

Chart Types

1. Posterior Charts: Bar charts showing metric distributions
2. Quality Radar: Radar charts for multi-dimensional quality
3. Risk Distribution: Doughnut charts for risk breakdown
4. Trend Analysis: Line charts for quality over time

🌐 API Reference

BayesianAIStats Class

Constructor

BayesianAIStats(array $initialData = [], ?string $storageFile = null)

Methods

addTrainingData(array $dataPoint): void

Add training data to update posterior distributions.

getPosteriors(): array

Get current posterior distributions.

savePosteriors(): void

Save posterior distributions to storage.

calculatePredictabilityScore(array $metrics): PredictabilityResult

Calculate Bayesian predictability score for given metrics.

exportStatistics(): array

Export current analysis statistics.

AICodeMetricsAnalyzer Class

Constructor

AICodeMetricsAnalyzer(array $codeData = [], array $weights = [], ?string $storageFile = null, ?array $config = [])

Parameters:

• $codeData: Sample data for context extraction
• $weights: Metric weights for analysis
• $storageFile: File for persisting Bayesian posteriors
• $config: Configuration for prompt/context sources

Methods

analyzeCodeQuality(array $codeMetrics, array $context = []): CodeAnalysisResult

Analyze code quality with optional prompt and generation context.

Parameters:

• $codeMetrics: Array containing raw_code, optional prompt, and optional generation_context
• $context: Additional context for analysis (project info, framework, etc.)

Returns: CodeAnalysisResult with comprehensive analysis including context

analyzeCodeWithPrompt(string $code, string $prompt, array $generationContext = []): CodeAnalysisResult

Convenience method for analyzing code with explicit prompt and context.

Parameters:

• $code: Raw code string to analyze
• $prompt: Original prompt or task description
• $generationContext: Array with developer experience, domain knowledge, etc.

Returns: CodeAnalysisResult with enhanced context-aware analysis

PromptContextManager Class

Constructor

PromptContextManager(array $config = [])

Parameters:

• $config: Configuration for prompt/context sources

Methods

extractPrompt(array $sample): ?string

Extract prompt from configured sources in priority order.

Parameters:

• $sample: Sample data containing file, content, and metadata

Returns: Extracted prompt string or null if no source available

extractGenerationContext(array $sample): array

Extract generation context from configured sources.

Parameters:

• $sample: Sample data containing file, content, and metadata

Returns: Array containing generation context (developer experience, domain knowledge, etc.)

CodeQualityTool Class

Constructor

CodeQualityTool(array $config = [], ?string $projectRoot = null)

Parameters:

• $config: Tool configuration (PHPStan, PHPCS, security tools, etc.)
• $projectRoot: Root directory for analysis (defaults to current working directory)

Methods

analyzeCode(string $code, ?string $filePath = null): array

Analyze code using configured real tools.

Parameters:

• $code: Source code to analyze
• $filePath: Optional file path for context

Returns: Array containing:

• success: Overall success status
• tools_run: List of tools that were executed
• metrics: Aggregated quality metrics (0-1 scale)
• issues: Detailed issues from each tool
• summary: Summary of each tool's execution

Supported Tools:

• PHPStan: Static analysis (syntax_correctness, logical_correctness, maintainability)
• PHPCS: Coding standards (syntax_correctness, maintainability, documentation_quality)
• PHPMD: Mess detection (maintainability, performance_optimization, logical_correctness)
• PHPUnit: Test coverage (test_coverage, logical_correctness, maintainability)
• Psalm: Security and type checking (security_compliance, syntax_correctness)
• Phan: Static security analysis (security_compliance, syntax_correctness)
• Security Checker: Dependency vulnerability scanning
• Composer Audit: Built-in dependency security analysis
• Custom Tools: Any CLI tool with flexible output parsing

CodeAnalysisResult Class

Methods

getContext(): array

Get the complete context including prompt and generation_context.

getPredictabilityResult(): PredictabilityResult

Get the Bayesian analysis results.

isProductionReady(): bool

Check if code meets production quality standards.

needsImmediateReview(): bool

Check if code requires immediate attention.

📁 Project Structure

besian-ai-stats/
├── src/
│   ├── BayesianAIStats.php           # Main analysis class
│   ├── BetaDistribution.php       # Beta distribution implementation
│   ├── PredictabilityResult.php   # Result container
│   ├── CodeAnalysisResult.php     # Complete analysis result
│   ├── AICodeMetricsAnalyzer.php  # AI code analysis with context
│   ├── PromptContextManager.php   # Configurable prompt/context extraction
│   ├── CodeQualityTool.php       # Real tool integration
│   ├── GitCodeSampler.php         # Git-based code sampling
│   └── CodeQualityToolSimulator.php # Legacy simulation (deprecated)
├── examples/
│   ├── git_training_generator.php   # Git-based training generator
│   ├── test_prompt_context_flow.php # Verification script
│   └── basic_usage.php           # Simple usage examples
├── tests/
│   ├── PromptContextPropagationTest.php # Basic functionality tests
│   ├── ConfigurablePromptContextTest.php # Configurable sources tests
│   └── ConcreteCodeQualityToolTest.php # Real tool integration tests
├── config.example.json            # Example configuration with all options
└── README.md                   # This file

🔒 License

GPLv2-or-later License - see LICENSE file for details

📞 Support

• Issues: GitLab Issues

Version: 0.1.0
Last Updated: 2026-02-09