mserajnik/szuruporter

A simple tool to mass-import media into szurubooru

1.10.0 2018-03-25 15:56 UTC

README

A simple tool to mass-import media into szurubooru

szuruporter is a simple command-line tool to mass-import media into szurubooru via its API. It's designed to work with hydrus network exports, but can also be used manually. Another great tool to combine this with is boorutagparser-server. You might also want to check out quicktagr, a tool to quickly apply tags to media files in the form of tags files (which is what hydrus network and szuruporter use) and a great companion to szuruporter.

The features are:

  • Importing media into szurubooru together with tags and safety rating
  • Updating existing posts with new tags and removing old ones (when adding an already existing media file to the import)
  • Updating the safety rating of existing posts (when adding an already existing media file to the import)
  • Automatically creating new tag categories or updating existing ones with configurable colors (when provided alongside of media files)
  • Assigning new tag categories to existing tags (when provided alongside of media files)

This is just what szuruporter can do. You don't have to import media together with tags and you also don't have to change something about existing posts, tags or tag categories when importing. But the options are there so you can easily modify data without having to do it manually multiple times.

E.g., if you are using hydrus network to manage your media (that you also want to have available in szurubooru) and want to change the tags of a file/post, you could just do it in hydrus network, export it together with the updated tags and use szuruporter to update it in szurubooru as well.

Possible future features will be:

  • Exporting posts alongside tags and tag categories (in the format used for hydrus network imports)
  • Deleting existing posts if the associated media files are not included in the import to allow szurubooru to operate in a "synced" state

Table of contents

Install

For ease of use, szuruporter is available on Packagist and intended to be installed globally via Composer.

user@local:~$ composer global require mserajnik/szuruporter

If you know your way around Composer you can do it in other ways as well but the rest of this readme will assume you've installed it globally.

Note: You might need to add Composer's global bin directory to your path. To do so in Linux and macOS, simply add the following to your .bashrc, .zshrc or whatever configuration you are auto-loading on login:

export PATH=~/.composer/vendor/bin:$PATH

Dependencies

Updating

To update szuruporter, simply use Composer:

user@local:~$ composer global update mserajnik/szuruporter

Usage

Use szuruporter at your own risks. It provides no way of rolling back changes and imports might only finish halfway if something goes wrong (resulting in incomplete and/or mangled data, depending on the settings). Backing up your data before importing is therefore strongly recommended.

CLI

To start off, you'll need to create a directory to import from (the provided directory path can be relative or absolute):

user@local:~$ szuruporter create ./some/path/

This will create a new directory called szurubooru-import (you can rename it afterwards) in the provided directory path and put a file called import.json there – this is the configuration file szuruporter needs to work. Every time you run an import, it will read the settings from this file.

You can have as many import directories as you want and all of them can have different settings in import.json – this makes it easy to handle imports for different szurubooru installations or to use different settings depending on what you are importing.

Next you'll need to adjust the settings in import.json – see Settings for detailed instructions on how to do so.

Note: Should a future update break an existing import.json, szuruporter will notify you and won't attempt to run using that import directory unless you update the settings. In that case, depending on the amount of changes, it might be faster to just create a new import directory and adjust the settings in that up-to-date import.json.

After you've adjusted the settings, put the media files you wish to import (along with the corresponding tags files – see Importing with tags for details) into the import directory and run szuruporter (again, the path can be relative or absolute):

user@local:~$ szuruporter run ./some/path/szurubooru-import/

Settings

szuruporter allows customization via the following settings:

version

Used to indicate if an outdated version of import.json is being used. Do not change this manually unless you know what you're doing.

  • Type: Integer
  • Default: 2

apiUrl

The API URL for the szurubooru installation you want to import into. Make sure there is no trailing /.

  • Type: String
  • Example: https://api.example.com

auth

username

The username for logging in to szurubooru.

  • Type: String
  • Example: username
password

The password for logging in to szurubooru.

  • Type: String
  • Example: password

logLevel

Changes what is logged to the console.

  • Type: Integer
  • Default: 1
  • Options:
    • 0: Logs errors only.
    • 1: Logs errors and infos.

safety

default

The default safety rating used when there is no rating:name tag provided.

  • Type: String
  • Default: safe
  • Options:
    • safe
    • sketchy
    • unsafe
ratings

Used for mapping safety ratings provided via rating:xyz tag to szurubooru's system.

  • Type: Object
  • Defaults:
    • explicit => unsafe
    • questionable => sketchy
    • safe => safe
  • Options: Each key/value pair represents a new mapping. The key represents the rating provided via rating:name, the value the needed rating for szurubooru's system. You can add as many mappings as you want.

posts

updateTags

Controls if already existing posts will be updated with new tags if the associated media file of the respective post is included in the import and any new tags are found within the tags file provided alongside of it.

  • Type: Boolean
  • Default: true
  • Options:
    • true
    • false
removeOldTags

Controls if already existing posts will have their existing tags removed if the associated media file of the respective post is included in the import and no tags file is provided alongside or the tags file doesn't contain the respective tags. Doesn't work when updateTags is set to false.

  • Type: Boolean
  • Default: false
  • Options:
    • true
    • false
updateSafety

Controls if already existing posts will have their safety rating updated with a new one. This will also overwrite the existing rating with the one set in safety.default if no rating:xyz tag is provided or a matching mapping doesn't exist.

  • Type: Boolean
  • Default: true
  • Options:
    • true
    • false

tagCategories

strip (tagCategories)

Any string provided in this array will be stripped from the tag category name. This should be matched with szurubooru's regex setting in order to not produce errors when importing. You can add as many strings as you want.

  • Type: Array
  • Defaults:
    • %
    • +
    • #
    • /
  • Options: An array with string values that are to be stripped.
defaultColor

The default color used when importing new tag categories and a matching mapping doesn't exist in tagCategories.colors.

  • Type: String
  • Default: #000000
  • Options: Any color in a hexadecimal format.
colors

Used for mapping tag categories to colors.

  • Type: Object
  • Defaults:
    • character => #00aa00
    • creator => #aa0000
    • medium => #aaaaaa
    • series => #aa00aa
  • Options: Each key/value pair represents a new mapping. The key contains the name of the tag category, the value the color in hexadecimal format. You can add as many mappings as you want.
updateColors

Controls if already existing tag categories will be updated with new colors if the respective tag category is included in one or more of the tags files when importing.

  • Type: Boolean
  • Default: true
  • Options:
    • true
    • false

tags

strip (tags)

Any string provided in this array will be stripped from the tag name. This should be matched with szurubooru's regex setting in order to not produce errors when importing. You can add as many strings as you want.

  • Type: Array
  • Default: No values
  • Options: An array with string values that are to be stripped.
updateTagCategories

Controls if already existing tags will have their tag categories updated if a new one is provided in the form of category:tag in one or more of the tags files when importing. Providing a tag without category will not remove an existing category from the tag.

  • Type: Boolean
  • Default: true
  • Options:
    • true
    • false

supportedExtensions

Contains the supported file extensions when importing. This should always match the extensions supported by szurubooru to not produce errors when importing.

  • Type: Array
  • Defaults:
    • gif
    • jpeg
    • jpg
    • mp4
    • png
    • swf
    • webm
  • Options: An array with string values, one per file extension.

Importing with tags

To import a media file alongside it's tags, simply put a tags file in the import directory that has the media file's name but with an added .txt extension. E.g., if you have a media file called lorem_ipsum.png, the tags file must be called lorem_ipsum.png.txt.

Inside a tags file, every tag must be on its own line. Like so:

car
clouds
scenery
sky

To automatically add a new tag category to a tag, you can provide it in the form category:tag:

brand:bmw
color:red
car
clouds
scenery
sky

If you have a tag with multiple :, the first one will define the position of the split by default. E.g., brand:bmw:m5 will cause the tag category to be brand and the tag to be bmw:m5.

To change this behavior, you can escape a : with \ and it will be ignored. This allows you to shift the split position. E.g., brand\:bmw:m5 will cause the tag category to be brand:bmw and the tag to be m5. If every occurence of : is escaped, no tag category will be added to the tag. E.g., brand\:bmw\:m5 will cause the tag to be brand:bmw:m5 without a tag category.

You can also provide a safety rating for a post by using the special prefix rating:

rating:safe
brand:bmw
color:red
car
clouds
scenery
sky

The suffix (in this example safe) can be anything you want, you just have to map it to szurubooru's rating system via the configuration in import.json. The rating suffix is a special case and doesn't add a tag safe with a category rating to a post like it would normally happen.

Any unwanted characters or strings can be removed from a tag or tag category by adjusting the configuration in import.json. E.g., prefix%:tag!+ can automatically be transformed into prefix:tag if you exclude % in tagCategories.strip and ! and + in tags.strip in import.json.

In addition to these configurable replacements, spaces will always be replaced with _ and the tags and tag categories will be trimmed of any whitespace.

hydrus network and boorutagparser-server will automatically provide the needed structure for the tags files and the tags inside of it and are therefore great choices if you don't want to create the files by hand.

Donate

If you like szuruporter and want to buy me a coffee, feel free to donate via PayPal:

Donate via PayPal

Alternatively, you can also send me BTC:

Donate BTC
13jRyroNn8QF4mbGZxKS6mR3PsxjYTsGsu

Donations are unnecessary, but very much appreciated. :)

Maintainer

mserajnik

Contribute

You are welcome to help out!

Open an issue or submit a pull request.

License

MIT © Michael Serajnik