Search by

therealworld / tools-plugin

therealworld

Shared foundation for the-real-world OXID 7 modules: helpers for config, DB, files, (S)FTP, APIs, XML/CSV and strings, Monolog logging, Twig filters and contracts for page cache and form protection.

Package info

bitbucket.org/therealworld/tools-plugin

Homepage

Issues

Type:oxideshop-component

pkg:composer/therealworld/tools-plugin

Statistics

Installs: 18 087

Dependents: 18

Suggesters: 0

v3.17.0 2026-11-16 20:44 UTC

This package is auto-updated.

Last update: 2026-11-16 20:46:48 UTC


README

Deutsche Fassung: README.de.md

vendor-logo the-real-world.de

description

Shared foundation of all the-real-world OXID 7 modules: helpers for config, database, files, (S)FTP, HTTP APIs, XML/CSV and strings, Monolog logging, typed access to request, session, cookie and model data, Twig filters, and contracts (interfaces with no-op defaults) that let modules work together without depending on each other - page cache, form protection, shipping data, return policy, visitor consent.

  • composer package: therealworld/tools-plugin
  • composer type: oxideshop-component - loaded by composer, no OXID module: there is no metadata.php, nothing to activate, and its services.yaml is always loaded.
  • own database table: trwstate (state store of ToolsState), created by a migration of the package - see migrations.
  • namespace: TheRealWorld\ToolsPlugin\

For the shop operator there is nothing to configure: the package is installed automatically as a dependency of the modules that need it. The rest of this document is aimed at developers.

requirements and installation

  • OXID eShop CE >= 7.4
  • PHP ^8.3 with ext-json, ext-curl, ext-zlib, ext-iconv, ext-simplexml, ext-xmlreader, ext-xmlwriter
  • libraries: guzzlehttp/guzzle ^7, fale/isbn, html2text/html2text, mobiledetect/mobiledetectlib, viison/address-splitter
composer require therealworld/tools-plugin

Optional - only needed for the respective feature and checked at runtime (composer.json suggest):

Extension / packageNeeded byRuntime check
ext-ftpToolsFTP (FTP/FTPS)ToolsFTP::isConnectionAvailable()
ext-soapToolsApi::executeExternApiSoap()ToolsApi::isSoapAvailable()
ext-zipToolsFile::compressZipFile() / extractZipFile()ToolsFile::isZipAvailable()
ext-imagickToolsImageConverter: animated GIFs stay animated (animated WebP, scaled frame by frame)ToolsImageConverter::isImagickAvailable(); without it an animated GIF is delivered unchanged
ext-posixCommandOwnershipGuard (owner warning in console commands)function_exists('posix_geteuid'); without it the warning is skipped
phpseclib/phpseclib >= 4ToolsFTP::uploadFileSFTP()ToolsFTP::isSftpAvailable()

phpseclib is optional on purpose: version 4 installs under the namespace phpseclib4\ but with the same package name, so a hard requirement here would decide the version for the whole shop. The module that actually transfers by SFTP requires it.

After installing or updating, run the migrations (also part of trw:update:after):

vendor/bin/oe-console trw:db:migrate migrate

conventions

These rules apply to every module building on this library.

  • DB result keys are always lowercase. ToolsDB::fetchAllByQb() and ToolsDB::normaliseRows() (for getCp()->fetchAllAssociative()) return the column keys in lowercase, whatever the SELECT says. MySQL itself answers in mixed case: base tables as typed in the SELECT, oxv_* views and information_schema always uppercase. So: write columns lowercase in the SELECT, read $aRow['oxtitle'], and never read a raw $qb->execute()->fetchAssociative() without normalising. Never set PDO::ATTR_CASE on the shared connection. (For writing - INSERT/UPDATE/batchReplace()/batchUpsert() - the case does not matter.)
  • Typed access instead of mixed:
    • model fields: DataGetter (getTRWStringData(), getTRWIntData(), getTRWFloatData(), getTRWBoolData(), getTRWDateData() ...) instead of getFieldData()
    • request / session / cookies: RequestGetter, SessionGetter, CookieGetter (and their Static* variants)
    • single untyped values (getConfigParam(), getViewDataElement(), raw rows): ToolsCast
    • structures we do not control (API answers, webhooks, decoded JSON/XML, import payloads): ToolsArray::arrayCheck*() for arrays, ToolsProp::propCheck*() for objects - two levels via the optional sub-key, deeper by chaining arrayCheckArray() / propCheckObject(). The first argument is deliberately mixed, so the calls chain without is_array() ladders.
  • Module settings from static config classes: StaticModuleSettings (+ StaticModuleDebugLog) instead of per-module getter boilerplate; the class only implements getModuleId().
  • Services: new code uses constructor injection, $this->getService() (classes extending Core\Base) or ContainerFacade::get(). The ServiceContainer / StaticServiceContainer traits only wrap ContainerFacade and are kept for existing call sites.
  • Twig filters/functions live here, not in a module: an unknown filter aborts the whole template, and only this package is guaranteed to be present.
  • The DB connection only via ToolsDB::getCp() - never ask the container for a core connection service yourself. getCp() is the one place to adapt when the core changes it (it now uses ConnectionFactoryInterface, the successor of the deprecated ConnectionProviderInterface).
  • Runtime state belongs into ToolsState, never into module settings or oxconfig. State is what the code sets for itself (locks, run anchors, counters, API tokens) - nobody configures it. Every save of a module setting makes the OXID core drop the complete template cache of the shop and rewrites the module's yaml; oxconfig holds the configuration of shop and themes. A getter must not write a setting back either: fall back to the default at runtime.

overview: core classes (src/Core/)

Almost all of them are static helper classes.

ClassPurpose
ToolsDBDB helpers: QueryBuilder/connection access (getQb(), getCp() - the shop's DBAL connection via ConnectionFactoryInterface), lowercase result fetch (fetchAllByQb(), fetchOneByQb(), fetchIntByQb(), normaliseRows()), quoting, batchReplace() / batchUpsert() (rows with different column sets are written in one statement per set, so a short row never blanks or overwrites columns it does not carry), buildSingleTableSql(), schema helpers (columns, indexes, field sizes, multilang/multishop registration), view names
ToolsStatestate store (table trwstate, one row per shop, owner and key): getString/Int/Bool/Array(), has(), set(), delete(), deleteModule(); locks with acquireLock(module, key, staleAfter) (atomic: one conditional upsert, two processes never both win; a lock older than staleAfter seconds may be taken over), releaseLock(), isLocked(), getLockSince(). Before the migration ran, reads return the default and writes do nothing (logged once)
SqlExpressionmarks a raw SQL value (NOW(), UUID() ...) that ToolsDB::buildSingleTableSql() must not quote
ListQueryConditionBuilderturns the condition arrays of the loadAll* list queries into a SQL fragment
ToolsConfigshop config access: typed config params, saving options, aarr/constraint options of shop and DB config, multishop/shop ids, getModuleExists(), legacy metadata events, config export; uploadFile() with an extension whitelist (default UPLOAD_EXTENSIONS_DEFAULT; php/phtml/phar/.htaccess & co. are always rejected, also hidden inside the name)
ToolsCastscalar-safe casts for single mixed values (castString(), castInt(), castStringList() ...)
ToolsArraytyped access to foreign array structures (arrayCheckString/Int/Float/Bool/Array/Object/List/StringList())
ToolsProptyped access to foreign object structures (propCheckString/Int/Float/Bool/Array/Object())
ToolsApiHTTP requests via Guzzle, cURL, file_get_contents or SOAP, with debug log; credentials (passwords, secrets, tokens, API keys, Authorization headers, URL userinfo, also inside JSON/XML/form bodies) are masked as *** before they reach the log (maskSensitiveData()/maskSensitiveString()); secrets without a telling name (e.g. an API key in a URL path) can be registered via addMaskValue(). If executeExternApiGuzzle() or executeExternApiCurl() gets no answer at all (DNS, connection, TLS, timeout), the result also holds errorType (timeout, connect or request) and errorMessage, so the caller can tell the merchant what went wrong; the cURL debug entry then also carries ApiCurlErrNo and ApiCurlError
ToolsFTPFTP/FTPS upload and sync, SFTP upload (phpseclib)
ToolsFilefile and directory operations: move (moveFile() returns whether the file arrived, moveFiles() false as soon as one found file fails — but true if no file matched), delete, create paths, gzip/zip, permission checks (checkPathPermissions() creates a missing path silently and only logs an error if that fails), image/media file validation, readable file type names for messages (getFileTypeLabel(): "PDF", "HEIC" ...), article image cleanup
ToolsDiskfree-disk-space guard for writers of many files; fails open when it cannot measure
ToolsHtaccessreads/writes the shop's .htaccess; fenced blocks owned by one module each
ToolsXMLXML reading/writing (array ↔ XML, streamed item reading via XMLReader)
ToolsExportCSV / ToolsExportXMLinstance-based CSV / XML export writers; ToolsExportCSV::writeRows() writes rows given as arrays (e.g. data built in memory instead of DB lists) with the same treatment (encoding/BOM, header, replacements, lengths, control characters) into any stream; options bEncloseAll (every field enclosed, numbers and empty ones too) and sEol (line ending, e.g. "\r\n") apply to both ways of writing
ToolsYamlYAML read/write
ToolsStringstring helpers: ASCII/normalisation, HTML → text, HTML sanitising, GTIN/ISBN cleanup, ISBN-aware search term groups, e-mail/date validation, UTF-8 handling, formatting of prices, durations, counts, bytes
ToolsAddresssplits a street line into street and house number (viison/address-splitter)
ToolsCountryhome country id, ISO alpha-2 code, country id by name
ToolsBrowseruser agent checks on top of MobileDetect: mobile, tablet, in-app browser
ToolsLangtranslation of language idents
ToolsLogin-memory error/info/debug log buffer used by the other helpers
ToolsMonologLoggerMonolog file loggers; flushToModuleFile() writes the ToolsLog buffer to log/<moduleId>/YYYY-MM-DD.log
ToolsMonologDBLoggerMonolog logger writing into a DB table (handler: Vendor/Monolog/OxidDBHandler, table given by the caller)
ToolsCacheclears shop caches: compile dir (tmp files, container, module and Twig template cache), page cache (via PageCachePurgerInterface), generated images, views, dynamic SEO urls; releaseBatchMemory() for long-running jobs (exports, feeds, sitemaps): call it after every batch - it resets the static per-process caches of the core models (Article, SeoEncoder, SeoEncoderArticle, ~27 KB per article otherwise) and switches the SEO file cache off for the rest of the process (in memory only); never in a normal request
ToolsSeocreate/delete static SEO urls, update SEO meta data
ToolsOxidOXID helpers: active user, admin checks, category path, writing attributes, manufacturers, category assignments and multilingual content; setCategoryTitle() reuses a category with the same title below the same parent instead of creating duplicates; setMultiLanguageContent() can create CMS pages inactive (last parameter $bActive)
ToolsThemeread/write/delete theme options
ToolsModuleChaininspects the class extension chain: is a method already implemented by another module, safe parent call with fallback, chain debugging
ToolsComposerVersionversion and path of a module or theme from its composer data (used for 'version' in metadata.php)
ToolsDynamicImageGeneratorscales an image file into another; configuration-free base of therealworld/dynimg-module. With blConvertImagesToWebP it makes the WebP master itself (ToolsImageConverter) instead of leaving it to the core - under a lock in trwstate, so parallel requests convert a large GIF once and wait for it -, and removes generated images of 0 bytes so they are made again. Animated GIFs/WebPs are scaled frame by frame with Imagick and copied unchanged without it (GD would keep only the first frame)
ToolsImageConverterJPEG/PNG/GIF → WebP (convertToWebp()), replacement for the core conversion, which writes 0 bytes for palette images (every GIF, many PNGs); an animated GIF becomes an animated WebP where Imagick is there. resizeAnimated() scales animated GIFs/WebPs with Imagick, isAnimatedGif()/isAnimatedWebp()/isAnimated() look at the first bytes only, fitSize() computes the size like the core. Imagick can be forbidden per process (setImagickAllowed(false) - for a module setting and to test the fallback). Writes atomically (temp file + rename), so a parallel request never reads a half written file

overview: traits (src/Traits/)

TraitPurpose
DataGettertyped model field access: getTRWStringData(), getTRWIntData(), getTRWFloatData(), getTRWBoolData(), getTRWArrayData(), getTRWRawStringData(), getTRWDateData(), getTRWDateString(), getTRWDateTimestamp(), hasTRWValidDate()
RequestGetter / StaticRequestGettertyped request parameters (getTrwStringRequestData(), getTrwStringRequestEscapedData(), getTrwIntegerRequestData() ...)
SessionGetter / StaticSessionGettertyped session variables, set/delete, ensureTrwSessionStarted(); getTrwSession() always returns the session the registry currently holds (no cached copy, safe across registry resets)
CookieGetter / StaticCookieGettertyped cookie read, set and delete
Json / StaticJsonJSON decode/encode, JSON POST body, JSON response
ServiceContainer / StaticServiceContainergetServiceFromContainer() / getStaticServiceFromContainer() - thin wrappers around ContainerFacade::get()
StaticModuleSettingstyped module settings for static *Config classes (getBoolSettingValue() ... getStringListSettingValue())
StaticModuleDebugLogcached isDebugLog() flag on top of StaticModuleSettings
ViewConfigTraitshared getters for ViewConfig extensions
LongRunningCommandconsole commands over large lists: pacing, clean interruption, progress for terminal and log, typed options
CommandOwnershipGuardwarns when a console command writes into a directory owned by another user (usually the web server); needs ext-posix, stays quiet without

overview: services (src/Service/, services.yaml)

All classes in src/Service/ are registered public and autowired under their class name.

ServicePurpose
OxNewServiceoxNew() wrapper, friendlier to static analysis
ModuleServicemodule ids/paths (all, active, inactive), typed module setting values, module version, read/save module configuration
CacheServiceresets template, module, language and menu cache
ClassExtensionChainServicereorders the class extension chain: move a module's extensions to the end or after another module, sort by dependency
TemplateExtensionChainServicemanages template_extension_chain.yaml: add (always on top = highest priority), remove, apply an order, sort
ToolsPluginMigrationPathProviderpoints to the migrations of this package (migration/migrations.yml), see migrations
ListLoaderServicebatch loading of whole lists (articles, categories, contents, manufacturers, vendors, orders) for exports, feeds, sitemaps; with a limit the queries are sorted stably (by oxid, or oxid as tie-breaker after the sort field), so consecutive batches neither repeat nor skip rows

In addition, services.yaml registers the core helpers under the ids trw.tools.api, trw.tools.cache, trw.tools.config, trw.tools.DB, trw.tools.disk, trw.tools.file, trw.tools.log, trw.tools.monologlogger, trw.tools.oxid, trw.tools.seo, trw.tools.string, trw.tools.xml. Since these classes are static, calling them directly is the usual way.

overview: contracts (src/Contract/)

Each interface is bound to a no-op default (Service/Null*), so a consumer always gets a usable service. A module joins in by decorating the interface in its own services.yaml; because a module's services.yaml is only loaded while the module is active, its contribution leaves again on deactivation. Chain interfaces expect each implementation to combine its result with the one of its $inner service instead of replacing it.

InterfaceDefaultPurpose
FormGuardInterfaceNullFormGuard (everything passes)form protection: isActive(), render(), verify() → FormGuardResult / FormGuardFailure; form keys in FormKey (user, newsletter, contact, details, password, suggest, withdrawal)
CacheKeyCookieContributorInterfaceNullCacheKeyCookieContributorcookie-derived fingerprint for a static cache key (getFingerprint())
CacheKeyParamContributorInterfaceNullCacheKeyParamContributorquery parameters that must stay in a static cache key (getAllowedParams()); base class AbstractCacheKeyParamContributor
CacheExcludedControllerContributorInterfaceNullCacheExcludedControllerContributorfrontend controllers a page cache must never serve (getExcludedControllers()); base class AbstractCacheExcludedControllerContributor
PageCacheDeliveryContributorInterfaceNullPageCacheDeliveryContributor (content unchanged)refreshes time or visitor bound markup each time a page cache hands out a stored page (prepareForDelivery()), e.g. a captcha's signed render timestamp; base class AbstractPageCacheDeliveryContributor
PageCacheProbeInterfaceNullPageCacheProbe (isWarm() = false)lets a warmup ask a full page cache whether a url is cached, its key, usage (PageCacheUsage)
PageCachePurgerInterfaceNullPageCachePurgerempties a full page cache (purge()); kept apart from the read-only probe
ArticleDeliveryPriceProviderInterfaceNullArticleDeliveryPriceProvider (null)shipping cost of an article to a country, e.g. for JSON-LD and feeds
ArticleDeliveryTimeProviderInterfaceNullArticleDeliveryTimeProvider (null)delivery time (DeliveryTime, business days)
MerchantReturnPolicyProviderInterfaceNullMerchantReturnPolicyProvider (null)the shop's return policy (MerchantReturnPolicy) for JSON-LD
ConsentProviderInterfaceNullConsentProvider (consent not required, everything allowed)visitor consent for tracking/marketing code: isConsentRequired() (is a consent manager in use?), hasConsent(ident, category) (category fallback via CATEGORY_NECESSARY/PERFORMANCE/FUNCTIONAL/ADVERTISING); replaced by a consent module (e.g. trwcookieconsent), asked by tracking modules (e.g. trwmatomo)

Example - a module bringing its own cache-relevant query parameter:

final class MyParamContributor extends AbstractCacheKeyParamContributor
{
    protected function getOwnParams(): array
    {
        return ['myparam'];
    }
}
services:
  My\Module\Service\MyParamContributor:
    decorates: TheRealWorld\ToolsPlugin\Contract\CacheKeyParamContributorInterface
    arguments:
      $inner: '@.inner'

migrations

OXID 7.4/7.5 only runs migrations of editions, the project and modules - not of components. OXID 7.6 adds that: oe:database:migrate also runs every service tagged oxid_esales.migration_path_provider (core MigrationPathProviderInterface). Until then this package bridges the gap:

  • Contract\MigrationPathProviderInterface - same contract as the core interface of 7.6: getMigrationConfigPath() returns the path of a Doctrine migrations.yml.
  • Register the provider with both tags. trw:db:migrate of therealworld/clirun-plugin runs ours, oe:database:migrate of OXID 7.6 the core one - the provider needs no change on the upgrade:
services:
  My\Component\Service\MyMigrationPathProvider:
    tags:
      - { name: 'trw.tools.migration_path_provider' }
      - { name: 'oxid_esales.migration_path_provider' }
  • Name the migration table after the OXID pattern oxmigrations_<suite id>; the suite id is what trw:db:migrate --project <suite id> selects. This package: oxmigrations_trwtoolsplugin (trwtools is the suite of the tools-module).

Twig extensions (src/Twig/Extensions/)

NameTypePurpose
trw_form_guard(formKey)functionrenders the protection markup of the FormGuardInterface chain ('' if nothing protects the form)
trw_form_guard_active(formKey)functiontrue if any protection module is active for the form
method_existsfilter{% if oView\|method_exists('getPageHeaderSlider') %}
trwtruncatefiltershortens text without leaking HTML entities (unlike the core truncate); never combine with \|raw
addhyphenstoisbnfilterhyphenates an ISBN (9783423117371 → 978-3-423-11737-1); other values pass unchanged

Twig template chain (core fix)

services.yaml decorates the two TemplateChainBuilder services of oxid-esales/twig-component (…shop_template_chain_builder, …modules_template_chain_builder) with Twig\ModeAwareTemplateChainBuilder: same arguments, same behaviour, but the template directories are cached per mode (admin / frontend). The core builder keeps the directories of the first mode for the whole request, also across an AdminModeChangedEvent. A mail with frontend templates sent from the admin (invoice, payment reminder, PayPal refund ...) then failed with "Error building inheritance chain for the template @__main__/email/html/header.html.twig" - but only when the chain cache was empty (after a cache clear or module refresh) and an admin template chain had been built earlier in the same request. On an update of twig-component, check the arguments against its services.yaml.

tests

tests/Unit/ holds PHPUnit tests (namespace TheRealWorld\ToolsPlugin\Tests\), currently for TemplateExtensionChainService, the upload file name rule of ToolsConfig, the credential masking of ToolsApi, the row grouping of the ToolsDB batch writers and the return values of ToolsFile::moveFile()/moveFiles().

bugs and issues

Please report bugs and issues at https://bitbucket.org/therealworld/tools-plugin/issues.