heimrichhannot / contao-entity-import-bundle
A backend only bundle. It offers data import from database or files to database.
Package info
github.com/heimrichhannot/contao-entity-import-bundle
Type:contao-bundle
pkg:composer/heimrichhannot/contao-entity-import-bundle
Requires
- php: ^8.1
- ausi/slug-generator: ^1.1
- contao/core-bundle: ^5.3
- guzzlehttp/guzzle: ^6.0 || ^7.0
- heimrichhannot/contao-be_explanation-bundle: ^2.5
- heimrichhannot/contao-field-value-copier-bundle: ^1.1
- heimrichhannot/contao-list-widget-bundle: ^1.0
- heimrichhannot/contao-multi-column-editor-bundle: ^2.4
- heimrichhannot/contao-utils-bundle: ^3.9
- league/oauth2-facebook: ^2.0
- symfony/cache: ^6.4
- symfony/config: ^6.4
- symfony/console: ^6.4
- symfony/dependency-injection: ^6.4
- symfony/event-dispatcher: ^6.4
- symfony/event-dispatcher-contracts: ^1||^2||^3
- symfony/http-kernel: ^6.4
- symfony/stopwatch: ^6.4
- terminal42/service-annotation-bundle: ^1.1
Requires (Dev)
- contao/contao-rector: dev-main
- contao/test-case: ^5.3
- rector/rector: ^2.2
Suggests
None
Provides
None
Conflicts
None
Replaces
None
- dev-main
- 0.25.2
- 0.25.1
- 0.25.0
- 0.24.x-dev
- 0.24.8
- 0.24.7
- 0.24.6
- 0.24.5
- 0.24.4
- 0.24.3
- 0.24.2
- 0.24.1
- 0.24.0
- 0.23.1
- 0.23.0
- 0.22.5
- 0.22.4
- 0.22.3
- 0.22.2
- 0.22.1
- 0.22.0
- 0.21.6
- 0.21.5
- 0.21.4
- 0.21.3
- 0.21.2
- 0.21.1
- 0.21.0
- 0.20.4
- 0.20.3
- 0.20.2
- 0.20.1
- 0.20.0
- 0.19.4
- 0.19.3
- 0.19.2
- 0.19.1
- 0.19.0
- 0.18.13
- 0.18.12
- 0.18.11
- 0.18.10
- 0.18.8
- 0.18.7
- 0.18.6
- 0.18.5
- 0.18.4
- 0.18.3
- 0.18.2
- 0.18.1
- 0.18.0
- 0.17.0
- 0.16.0
- 0.15.0
- 0.14.0
- 0.13.0
- 0.12.0
- 0.11.0
- 0.10.2
- 0.10.1
- 0.10.0
- 0.9.0
- 0.8.2
- 0.8.1
- 0.8.0
- 0.7.1
- 0.7.0
- 0.6.1
- 0.6.0
- 0.5.0
- 0.4.0
- 0.3.2
- 0.3.1
- 0.3.0
- 0.2.1
- 0.2.0
- 0.1.3
- 0.1.2
- 0.1.1
- 0.1.0
- dev-feature/instagram-login
This package is auto-updated.
Last update: 2026-11-10 09:16:26 UTC
README
This bundle offers a generic importer to migrate data from various sources to contao database entities.
This Bundle is still in development.
Features
- import data from either file content or database into arbitrary contao database entities (
tl_*) - support for various data types (json, csv, ...)
- support for various source types (contao file system, http, absolute path)
- social media sources: Instagram, Facebook pages, YouTube channels and LinkedIn company pages, with access token login and refresh in the back end
- executable from contao backend, as cronjob and symfony command
- automatic field manipulation (sorting, alias, dateAdded, tstamp)
- merge and insert mode
- email and contao_log notifications while executing import with exceptions (will be sent once per importer configuration, will be reset after next successful import)
- (optional) support for DC_Multilingual if installed
Impressions
Importer source backend settings:
Importer config backend settings:
Installation
Install via composer: composer require heimrichhannot/contao-entity-import-bundle and update your database.
Configuration
- Navigate to "Import" in the Contao backend in the section "system".
- Create an importer source to your needs.
- Create an importer using the source created in the step before.
- Run the importer either using dry-run or the normal mode.
config.yml
huh_entity_import: debug: contao_log: true # log errors while importing to contao system log email: false # report errors while importing via email to the admin email defined in the contao settings
Social media sources
Besides databases and files, an import source can read the posts of a social media account. Each source type has a fixed set of source fields; the field mapping offers them in a select (Instagram: type the field name).
| Type | Platform API | Credentials | Token lifetime |
|---|---|---|---|
instagram (Instagram Login) |
Instagram API with Instagram Login (graph.instagram.com) | access token | 60 days, refreshed automatically |
instagram (Facebook Login) |
Instagram API with Facebook Login (graph.facebook.com) | Meta app, page token via login | does not expire |
facebook |
Graph API /{page-id}/posts |
Meta app, page token via login | does not expire |
youtube |
YouTube Data API v3 | API key | – |
linkedin |
Posts API of LinkedIn's Community Management API | LinkedIn app, token via login | 60 days, refreshed for a year |
Answers of the APIs are cached for 5 minutes. Errors show up as back end messages with the API's error message (never with the request URL, which may contain the token).
Access tokens
- Request access token (Facebook, Instagram with Facebook Login, LinkedIn): save the app credentials first, then
click the button. You log in with the platform, grant the app access and return to the import source with the token
stored. The help text below the button shows the redirect URL to enter in the app's settings. The redirect URLs are
the ones contao-newsroom-bundle used (
/contao-newsroom/facebook/{id}/redirect_callback,/contao-newsroom/auth/linkedin), so apps set up for it keep working. - Refresh access token now (Instagram with Instagram Login, LinkedIn with a refresh token): tick the checkbox and the source is saved and its token exchanged for a new one with the full lifetime right away.
- A daily cron job (
AccessTokenRefreshCron) refreshes those tokens 30 days before they expire (or when the expiry is unknown). Failures go to the Contao system log. For LinkedIn it also logs an error every day during the last 30 days of the refresh token, since only a new login helps then.
- Instagram API with Instagram Login: for Instagram professional (business or creator) accounts. Generate a token
for the account in the Meta app's dashboard (Instagram > API setup with Instagram login) and paste it. Graph API version: e.g.
v24.0. Instagram allows a refresh once a token is at least 24 hours old. - Instagram API with Facebook Login: for Instagram professional accounts connected to a Facebook page. Enter the Meta app's ID and secret and the Graph API version, request the access token and log in with an account that manages the page. The token of the page connected to the Instagram account is stored; if several of your pages have one, enter the Instagram user ID first (the error message lists them).
- Source fields:
id,caption,media_type,media_url(the thumbnail for videos),thumbnail_url,permalink,timestamp(Unix time),username. Posts not shown in the profile's feed are skipped.
Facebook pages
- Create a Meta app (developers.facebook.com) that uses Facebook Login and may ask for the permissions
pages_show_listandpages_read_engagement(the login passes them asscope). Add the redirect URL shown below the "Request access token" button as "Valid OAuth Redirect URI". - As long as the account that logs in has a role on the app (administrator, developer or tester), standard access to these permissions suffices and no App Review is needed.
- Enter app ID, app secret, Graph API version (e.g.
v24.0) and, if you manage more than one page, the page ID; save and request the access token.
The page token does not expire, but stops working if the account that logged in loses its role on the page, changes its password or removes the app; request a new one then.
Source fields: id, message, full_picture, permalink_url, created_time (Unix time), status_type.
YouTube
Create an API key for the YouTube Data API v3 in the Google Cloud console. Find the channel by its ID (UC…), its
handle (@name) or, for old channels, its user name. The source reads the latest 50 public videos from the channel's
uploads playlist: 2 of the 10,000 daily quota units per import.
Source fields: id, title, description (characters beyond ASCII as numeric entities), publishedAt (Unix time),
channelId, channelTitle, image_url (the largest thumbnail), url.
LinkedIn company pages
Reading an organization's posts needs LinkedIn's Community Management API, which LinkedIn grants after reviewing the app and verifying the company (developer.linkedin.com > your app > Products). Without it the login fails with an unauthorized scope.
- In the app's "Auth" tab, add the redirect URL shown below the "Request access token" button.
- Enter client ID and secret, the organization ID (the number in the page's admin URL), the API version (
YYYYMM; LinkedIn supports each version for at least a year) and the number of posts; save and request the access token with an account that administers the page. - Access tokens last 60 days. Apps with programmatic refresh tokens also get a refresh token that lasts a year from the login, and the token is refreshed automatically until then.
Source fields: id (the post URN), url, commentary (plain text: mentions as names, hashtags as #tag),
publishedAt (Unix time), image_url (image or video thumbnail; only requested if mapped), article_url,
article_title. Drafts and sponsored posts that are not shown in the feed are skipped.
To store images, map image_url (or media_url, full_picture) to a file field in the importer's file field mapping.
Platforms not supported
X (Twitter), which contao-newsroom-bundle supported, is not: reading timelines needs paid API access.
Technical instructions
Output messages during import process respecting the context
An importer can be run as command, cronjob or in the web context. The context implies how to output messages (or error messages).
For example, when running an importer by clicking the "import" button in the website (i.e. in the browser), the messages should be
printed out using \Contao\Message::addError() (or addConfirmation(), ...). If the import is run as command, the same messages
need to be printed out on command line using a SymfonyStyle instance. If the command is also run in web context, the messages even
need to be stored to the tl_entity_import_config entity, because these are retrieved by ajax in the progress bar view.
As you can see, it's not just "printing a message", but taking care of the context, as well.
For this you can use the method ImporterInterface::outputResultMessage() which takes care of the correct context itself.
Run as symfony command
huh:entity-import:execute config-ids [--dry-run]
where config-ids needs to be a comma separated list of importer config IDs.
Arguments
| Argument | Mandatory | Type | Description |
|---|---|---|---|
| config-id | true | integer | The ID of the importer configuration |
| dry-run | false | boolean | Run importer without writing data into database |
Run as contao cron
Import is executable with contao poor man's cron. The interval of execution is similar to the contao definition.
The import configuration allows to enable cron execution and picking of the cron interval.
Possible to choose between minutely, hourly, daily, weekly, monthly interval. It is recommended to setup
the debug options in config.yml before importing via cronjob.
Add custom sources
- Create the source and extend from
\HeimrichHannot\EntityImportBundle\Source\AbstractSourceor implement the interface\HeimrichHannot\EntityImportBundle\Source\SourceInterface. - Create an event listener for the event
SourceFactoryCreateSourceEvent:class SourceFactoryCreateSourceEventListener { public function __invoke(SourceFactoryCreateSourceEvent $event) { $sourceModel = $event->getSourceModel(); $source = $event->getSource(); switch ($sourceModel->type) { case 'new_stuff': $source = new NewStuffSource(); break; // ... } $event->setSource($source); } }
- Create a
tl_entity_import_source.phpand add your sources in thetypefield's options:$dca = &$GLOBALS['TL_DCA']['tl_entity_import_source']; $dca['fields']['type']['options'] = array_merge($dca['fields']['type']['options'], [ 'new_stuff' ]);
- Create your palette in your
tl_entity_import_source.php, e.g. by copying one from this bundle.
Add presets for source/config field mapping
Adding a field mapping might be tedious sometimes. Hence we provide an event for adding presets which then are selectable in the backend entity (source and config).
Just register an event listener and add your presets the following way:
$presets = $event->getPresets();
$presets = array_merge($presets, [
'twitter' => [
[
'name' => 'externalId',
'valueType' => 'source_value',
'sourceValue' => 'id',
],
[
'name' => 'text',
'valueType' => 'source_value',
'sourceValue' => 'full_text',
],
[
'name' => 'publishedAt',
'valueType' => 'source_value',
'sourceValue' => 'created_at',
],
[
'name' => 'imageUrl',
'valueType' => 'source_value',
'sourceValue' => 'image_url',
],
[
'name' => 'imageId',
'valueType' => 'source_value',
'sourceValue' => 'image_id',
],
[
'name' => 'url',
'valueType' => 'source_value',
'sourceValue' => 'url',
],
]
]);
$event->setPresets($presets);
Add quick importers for editors
To be honest: A "quick importer" is a nicer term for "easy importer", so that editors don't get sad ;-) The use case for quick importers is to have a user interface which is way easier to use for editors.
Simply create an import source and import config as you would normally. Afterwards editors can create quick importers on their own and add custom file sources (currently only files of the type CSV are supported).
Run the import via cronjob in web context
If you have large datasets the import might take too much memory of the web server. In this case it's often useful to
- Run the importer as symfony command. This has to be done by a developer.
- Run the importer in the web context by cronjob. This can be done by editors or developers.
In most cases, running an importer by command on the shell takes way less memory compared to running it in the web context via website.
For option 2 to work you need to do the following steps:
- Add the bundle's import command as cronjob using the parameter
--web-cron-mode:
* * * * * /path/to/vendor/bin/contao-console huh:entity-import:execute <import config ids> --web-cron-mode >/dev/null 2>&1 - Activate
useCronin the importer config and fill out the needed fields. - Activate
useCronInWebContextin the importer config.
Events
| Name | Description |
|---|---|
AddConfigFieldMappingPresetsEvent |
Add config field mapping presets |
AddSourceFieldMappingPresetsEvent |
Add source field mapping presets |
AfterCsvFileSourceGetRowEvent |
Modify csv rows e.g. for fixing file encoding issues |
AfterFileSourceGetContentEvent |
Configure the data after receiving from source |
AfterImportEvent |
Get imported data after finished import |
AfterItemImportEvent |
Get imported item data after finished import |
BeforeAuthenticationEvent |
Configure authentication data before sending GET request to http source |
BeforeImportEvent |
Configure the data before importing |
BeforeItemImportEvent |
Configure the item data before importing; call setSkipped(true) in order to skip the import |
SourceFactoryCreateSourceEvent |
Implement custom logic for new custom file sources |
BeforeFileImportEvent |
Implement a custom logic for setting the folder and/or file path when importing files |
Development
Run the tests:
composer install vendor/bin/phpunit
The tests of the social media sources run against a mocked HTTP client; they need no credentials.


