README

Fao-Toolkit Console Application

Intended use for Gene banks

FAO Toolkit purpose is to be a terminal client toolkit to Gene banks, to make it more manageable to exchange data with FAO. The implemented features are the following at the moment:

Get registered institute information like FAO Institute code AKA WIEWS code and their addresses.
Register Accession order Standard Material Transfer Agreements (SMTA)
Register Multi Crop Passport Data (GLIS) to get DOI, and then keep it updated.

The client is generic implemented, and is configurable to be able to work with several Gene bank information systems. The idea is in the configuration file map the necessary table columns of the information system in use.

Getting started

First you need to require necessary database library that your Information system is using. Read more of this in the installation section.

To make SMTA posting/registration work, you need to have the following:

An account at FAO's Planttreaty site.
Two database tables or dataviews with SMTA and SMTA material data.
Make the configuration in this application's config file.
run ./yii (or bin/fao-toolkit) smta/register command. The command uses parameters to limit which SMTAs you are intended to post/register.

To register and/or updating DOIs, you need to have the following:

An account at FAO's Planttreaty site (It is really the same account as above. If you have an Easysmta account. You will also have an account to registry DOI's. Anyway contact the Office of International Treaty on Plant Genetic Resources for Food and Agriculture, if you represent a national Gene bank or similar.)
Two or more database tables or dataviews with GLIS data (https://www.fao.org/3/bt113e/bt113e.pdf). Read fao-params-dist.php for more information
Make the configuration in this application's config file.
run ./yii (or bin/fao-toolkit) glis:register command. The command uses parameters to limit which GLISs you are intended to post/register.

Installation

In order to install this project via composer run the following command in the console:

composer create-project endlessdreams/fao-toolkit=^3.0 fao-toolkit

or add the package manually to your composer.json file in the require section:

"endlessdreams/fao-toolkit": "^3.0"

or globally:

composer global create-project endlessdreams/fao-toolkit=^3.0 fao-toolkit

This will install fao-toolkit into the ~/.composer/vendor/ directory and, most importantly, the fao-toolkit CLI tool are installed into ~/.composer/vendor/bin/.

Simply add this directory to your PATH in your ~/.bash_profile (or ~/.bashrc) like this:

export PATH=~/.composer/vendor/bin:$PATH

and fao-toolkit is now available on your command line.

To keep your tools up to date, you simply do this:

composer global update
composer global install

To remove a package, you edit ~/.composer/composer.json and then run composer global update.

Database

To be able to use other database than SQLite, you need to install those packages and drivers too. Foa-Toolkit is tested on besides SQLite on the following database types; MySql, Postgres and SQL server.

To set up this command to use another database follow the link.

Configuration

Major configuration file called fao-params-dist.php is located in sub folder to fao-toolkit application config folder. The sub folder can be any of:

prod for production. This is the one you normally use.
dev for development to a fake server.
test which is used for the codeception tests

Do not change fao-params-dist.php file. Make a copy instead with the name fao-params-local.php

In fao-params-dist.php file you will see the following structure like:


return [
    'options' => [
        'version' => '3',
    ],
    'databases' => [
        '@defaultModule' => 'smta',
        'database' => [
            0 => [
                '@module' => 'smta',
                'driver' => $_ENV['FT_DB_DRIVER'] ?: 'sqlite',
                'host' => $_ENV['FT_DB_HOST'] ?: 'localhost',
                'database_name' => $_ENV['FT_DB_DATABASE'] ?: '@resources/database/smta-prod.sq3',
                'port' => $_ENV['FT_DB_PORT'] ?: null,
                'options' => $_ENV['FT_DB_OPTIONS'] ?: [],
                'username' => $_ENV['FT_DB_USERNAME'] ?: 'dbuser',
                'password' => $_ENV['FT_DB_PASSWORD'] ?: 'dbpassword',
            ],
        ],
    ],
    ...

To set database and FAO HTTPS API credentials, you could do it in either:

In the config file
Setting environment variables in the .env file
Or exporting environment variables in /etc/environment, an make sure that they won't be overridden in the .env-file.

For more information in how to configure the command, read more in configuring file.

Usage

The examples are written as if your working directory is bin or the application is globally installed. More usage examples will be found in the usage file.

The most important command is to list your options.

fao-toolkit help

Second most important command is list the available commands.

fao-toolkit list

Now done with the basics, now some real examples

To get information of WIEWS institute code registered organisation

fao-toolkit wiews:fetch NOR051

To get information of WIEWS institute code registered organisation and selecting output format like JSON.

fao-toolkit wiews:fetch --format=json -- NOR051

To test generate a GLIS (extended standard of MCPD) to terminal. As you probably note, that we use a "test-option switch". Keep it so you avoid making requests by accident to FAO production API. If you have configured your config with correct URL and credentials.

fao-toolkit glis:register --test --dry-run --limit=1 -vv

To test generate all glis xml with last updated from 2020-01-01 and forward. (No https credentials needed here, only to your database)

fao-toolkit glis:register --test --dry-run --from=2023-01-01 --limit=1000 --skip-invalid

You might note that an overview stat was printed out and even some validation errors. For every Glis or Smta command call an Excel-file with task stats will be created. The temp folder is the default folder of this file.

To choose another output path, e.g. "./stats_file.xlsx". This file can be a help, when there were som validation errors. We will then have them listed, and with a note of which validation was broken.

fao-toolkit glis:register --test --dry-run --from=2023-01-01 --limit=1000 --skip-invalid --output-path=./stats_file.xlsx

To test generate xml to terminal.

fao-toolkit smta:register --test --dry-run -vv

To test generate all smta xml to 2018-01-01. (No https credentials needed here, only to your database). Similar way as glis:register. Note that with no explicit order_by option, for smta the default order column is date (date of smta).

fao-toolkit smta:register --test --dry-run --to=2018-01-01

To test post all SMTA's until 2018-01-01.

fao-toolkit smta:register --to=2018-01-01 --test

To test post all SMTA's from 2017-01-01 until 2018-01-01.

fao-toolkit smta:register --from=2017-01-01 --to=2018-01-01 --test

To actual post all SMTA's until 2018-01-01. (No testing)

fao-toolkit smta:register --to=2018-01-01

If several gene banks are hosted on the same server, you have to explicit refer to which gene bank / institute you will register SMTA's. Then you use the run-as parameter with the WIEWS-code.

fao-toolkit smta:register --to=2018-01-01 --run-as=SWE054

If the run-as institute uses accession numbers with different prefixes and only one set should be registered. (Note: this is for a MS SQL server, otherwise Regular expressions will be used.)

fao-toolkit smta:register --to=2018-01-01 --run-as=SWE054 --filter="NGB %"

Testing

Unit testing

The template comes with ready to use Codeception configuration. In order to execute tests run:

vendor/bin/codecept run

Static analysis

The code is statically analyzed with Psalm. To run static analysis:

./vendor/bin/psalm

Credits

ir. R (Roel) Hoekstra and dr.ir. TJL (Theo) van Hintum at Centrum voor Genetische Bronnen Nederland for letting me look at their dos cmd toolkit.

Mr Marco Marsella at FAO, Climate, Biodiversity, Land and Water Department, IT for great support.

License

This command line program is licensed under AGPL.

Please see LICENSE for more information.

endlessdreams / fao-toolkit

Maintainers

Details