README

A powerful, production-ready semantic search package for Laravel that enables natural language querying across your Eloquent models with intelligent English-to-SQL translation. No embeddings, no ML training—just instant, accurate search powered by static code analysis.

✨ Features

🎯 Natural Language Queries - Query your models using plain English
🧠 Intelligent Code Analysis - Automatically analyzes your Laravel project structure
🌍 Multilingual Support - English, Spanish, French, German, Portuguese
🔍 Context-Aware Search - Understands business logic, relationships, and UI patterns
🔗 Relationship Intelligence - Navigates Eloquent relationships automatically
🚀 Zero Training Required - Works instantly with your existing codebase
💾 Multi-Database Support - MySQL, PostgreSQL, SQLite
🤖 LLM Fallback - Optional OpenAI integration for complex queries
🔒 Security First - Built-in authorization and access control
⚡ Performance Optimized - Intelligent caching and query optimization
📊 Aggregation Support - Count, sum, average, min, max with grouping
🔀 Boolean Logic - AND/OR operators in filters

📋 Requirements

PHP 8.1 or higher
Laravel 9.x, 10.x, or 11.x
MySQL 8.0+, PostgreSQL 12+, or SQLite 3.0+
Composer

📦 Installation

Install via Composer

composer require venmail/laravel-semantic-search

Publish Configuration

php artisan vendor:publish --tag="semantic-search-config"

This will create config/semantic-search.php in your Laravel application.

🚀 Quick Start

Basic Usage

use Venmail\SemanticSearch\Facades\SemanticSearch;

// Simple queries
$results = SemanticSearch::search("active users older than 25");
$results = SemanticSearch::search("users with posts created last month");

// With pagination
$results = SemanticSearch::search("high-value transactions")
    ->paginate(20);

// Aggregation queries
$results = SemanticSearch::search("count users by country");
$results = SemanticSearch::search("sum of transactions by category");

Example Controller

<?php

namespace App\Http\Controllers;

use Venmail\SemanticSearch\Facades\SemanticSearch;
use Illuminate\Http\Request;

class SearchController extends Controller
{
    public function search(Request $request)
    {
        $query = $request->input('q');
        
        $results = SemanticSearch::search($query, [
            'paginate' => 20,
            'locale' => app()->getLocale(),
        ]);
        
        return response()->json($results);
    }
}

🎯 Supported Query Types

Simple Entity Queries

"show users"
"list orders"
"display products"

Filtered Queries

"users older than 25"
"transactions over $1000"
"posts created last month"
"active users with email containing @example.com"

Relationship Queries

"users with posts"
"customers and their orders"
"products with reviews"
"users with posts that have comments"

Aggregation Queries

"count users"
"sum of transactions by category"
"average age of users"
"total revenue by month"
"count orders by status"

Complex Queries

"active users who made purchases over $500 in the last 30 days"
"customers with more than 5 orders and total spending over $1000"
"users with posts that have more than 100 likes"

Boolean Logic

"users with posts and comments" (AND)
"active users or premium users" (OR)

⚙️ Configuration

Basic Configuration

Edit config/semantic-search.php:

return [
    'analysis' => [
        'auto_discover' => true,
        'scan_directories' => [
            'controllers' => app_path('Http/Controllers'),
            'models' => app_path('Models'),
            'views' => resource_path('views'),
            'routes' => base_path('routes'),
        ],
    ],

    'cache' => [
        'enabled' => env('SEMANTIC_SEARCH_CACHE_ENABLED', true),
        'default_ttl' => env('SEMANTIC_SEARCH_CACHE_TTL', 3600),
        'prefix' => env('SEMANTIC_SEARCH_CACHE_PREFIX', 'semantic_search'),
    ],

    'llm' => [
        'enabled' => env('SEMANTIC_SEARCH_LLM_ENABLED', false),
        'provider' => env('SEMANTIC_SEARCH_LLM_PROVIDER', 'openai'),
        'model' => env('SEMANTIC_SEARCH_LLM_MODEL', 'gpt-3.5-turbo'),
        'api_key' => env('OPENAI_API_KEY'),
        'timeout' => env('SEMANTIC_SEARCH_LLM_TIMEOUT', 10),
        'confidence_threshold' => env('SEMANTIC_SEARCH_LLM_CONFIDENCE', 0.7),
        'cache_ttl' => env('SEMANTIC_SEARCH_LLM_CACHE_TTL', 3600),
    ],

    'security' => [
        'max_rows' => env('SEMANTIC_SEARCH_MAX_ROWS', 10000),
        'restricted_models' => env('SEMANTIC_SEARCH_RESTRICTED_MODELS', '') 
            ? explode(',', env('SEMANTIC_SEARCH_RESTRICTED_MODELS')) 
            : [],
        'restricted_fields' => env('SEMANTIC_SEARCH_RESTRICTED_FIELDS', '') 
            ? explode(',', env('SEMANTIC_SEARCH_RESTRICTED_FIELDS')) 
            : [],
    ],

    'disambiguation' => [
        'enabled' => env('SEMANTIC_SEARCH_DISAMBIGUATION_ENABLED', true),
        'confidence_threshold' => env('SEMANTIC_SEARCH_DISAMBIGUATION_CONFIDENCE', 0.6),
    ],

    'performance' => [
        'max_query_time' => env('SEMANTIC_SEARCH_MAX_QUERY_TIME', 30),
        'memory_limit' => env('SEMANTIC_SEARCH_MEMORY_LIMIT', '256M'),
    ],
];

Environment Variables

Add to your .env:

# Cache
SEMANTIC_SEARCH_CACHE_ENABLED=true
SEMANTIC_SEARCH_CACHE_TTL=3600

# LLM (Optional)
SEMANTIC_SEARCH_LLM_ENABLED=false
OPENAI_API_KEY=your_api_key_here
SEMANTIC_SEARCH_LLM_MODEL=gpt-3.5-turbo

# Security
SEMANTIC_SEARCH_MAX_ROWS=10000
SEMANTIC_SEARCH_RESTRICTED_MODELS=
SEMANTIC_SEARCH_RESTRICTED_FIELDS=

# Performance
SEMANTIC_SEARCH_MAX_QUERY_TIME=30

🌍 Multilingual Support

The package supports multiple languages out of the box:

// English (default)
SemanticSearch::search("show active users", ['locale' => 'en']);

// Spanish
SemanticSearch::search("mostrar usuarios activos", ['locale' => 'es']);

// French
SemanticSearch::search("afficher les utilisateurs actifs", ['locale' => 'fr']);

// German
SemanticSearch::search("aktive Benutzer anzeigen", ['locale' => 'de']);

// Portuguese
SemanticSearch::search("mostrar usuários ativos", ['locale' => 'pt']);

🤖 LLM Fallback (Optional)

For complex queries that the static parser cannot handle, enable LLM fallback:

SEMANTIC_SEARCH_LLM_ENABLED=true
OPENAI_API_KEY=your_api_key_here

The LLM fallback automatically activates when:

Query confidence is below threshold
Complex multi-clause queries
Ambiguous entity names
Domain-specific terminology

🔒 Security

Access Control

The package includes built-in security features:

// Restrict access to specific models
'restricted_models' => [
    'App\Models\SensitiveData',
    'App\Models\AdminUser',
],

// Restrict access to sensitive fields
'restricted_fields' => [
    'password',
    'secret_key',
    'credit_card',
],

Authorization

Queries are automatically checked against:

Model access restrictions
Field access restrictions
Relationship traversal permissions
Row limit constraints

📊 Advanced Usage

Custom Query Options

$results = SemanticSearch::search("users", [
    'paginate' => 20,        // Pagination
    'limit' => 100,          // Limit results
    'locale' => 'en',         // Language
]);

Query History

// Get user's search history
$history = SemanticSearch::getHistory(limit: 20);

// Get suggestions based on history
$suggestions = SemanticSearch::getSuggestions("user", limit: 5);

Cache Management

// Clear all cache
SemanticSearch::clearCache();

// Clear specific pattern
SemanticSearch::clearCache('users:*');

🏗️ Architecture

The package uses intelligent static code analysis to understand your Laravel application:

Project Analysis - Scans controllers, models, views, and routes
Vocabulary Building - Creates dynamic vocabulary from your codebase
Query Parsing - Parses natural language into structured queries
Disambiguation - Handles spelling, entities, and context
Query Building - Translates to optimized Eloquent queries
Execution - Runs queries with proper security and caching

🔧 Database Support

The package supports multiple database engines:

MySQL 8.0+ - Full support
PostgreSQL 12+ - Full support
SQLite 3.0+ - Full support

Database-specific optimizations are automatically applied.

📈 Performance

Caching Strategy

Metadata Cache - Project analysis results (4 hours)
Schema Cache - Database schema information (4 hours)
Query Cache - Search results (configurable, default 1 hour)
LLM Cache - LLM responses (configurable, default 1 hour)

Query Optimization

Index-aware filter ordering
Efficient date range queries
Optimized relationship joins
Proper GROUP BY handling

🐛 Troubleshooting

Common Issues

Query not parsing correctly:

Check that models exist and are accessible
Verify field names match database columns
Enable LLM fallback for complex queries

Slow queries:

Enable caching
Check database indexes
Reduce query complexity

Permission errors:

Check security configuration
Verify user authentication
Review restricted models/fields

📚 Examples

E-commerce Application

// Find products
SemanticSearch::search("laptops under $1000 with 16GB RAM");

// Customer analytics
SemanticSearch::search("customers who purchased in last 30 days");

// Sales reports
SemanticSearch::search("total sales by category this month");

Content Management

// Find content
SemanticSearch::search("published articles about Laravel");

// User management
SemanticSearch::search("active users with admin role");

// Analytics
SemanticSearch::search("count of posts by author");

🤝 Contributing

📜 Changelog

See CHANGELOG.md for release notes and upgrade guidance.

Contributions are welcome! Please read our contributing guidelines and submit pull requests.

📄 License

This package is open-sourced software licensed under the MIT license.

🙏 Acknowledgments

Laravel team for the excellent framework
OpenAI for LLM API
The Venmail team and early adopters

📞 Support

Documentation: See DEV.md for development setup
Issues: GitHub Issues

Made with ❤️ for the Laravel community

venmail / laravel-semantic-search

Maintainers

Package info

Statistics

Security