voilab / csv
CSV parser that uses `fgetcsv` to parse file, string, streams or arrays, extract columns and provide per-column method to manipulate data.
Installs: 89
Dependents: 0
Suggesters: 0
Security: 0
Stars: 0
Watchers: 4
Forks: 0
Open Issues: 2
Type:application
Requires
- php: >=7.1
- psr/http-message: ^1.0
Requires (Dev)
- guzzlehttp/guzzle: ^6.5
- phpunit/phpunit: ^7.5
- scrutinizer/ocular: ^1.5
- dev-develop
- 5.1.1
- 5.1.0
- 5.0.0
- 4.2.1
- 4.2.0
- 4.1.0
- 4.0.0
- 3.1.1
- 3.1.0
- 3.0.0
- 2.0.1
- 2.0.0
- 1.1.1
- 1.1.0
- 1.0.5
- 1.0.2
- 1.0.0
- 0.5.3
- 0.5.2
- 0.5.0
- 0.4.7
- 0.4.6
- 0.4.5
- 0.4.4
- 0.4.2
- 0.4.0
- 0.3.2
- 0.3.0
- 0.2.0
- dev-master
- dev-dependabot/composer/guzzlehttp/psr7-1.9.1
- dev-dependabot/composer/guzzlehttp/guzzle-6.5.8
- dev-feature/guess
- dev-feature/php5
This package is auto-updated.
Last update: 2024-10-16 12:36:15 UTC
README
This class uses fgetcsv
to parse a file or a string, extract columns and
provide per-column methods to manipulate data.
It can parse large files, HTTP streams, any types of resources, or strings.
It comes with a basic error handling, so it is possible to collect all errors in the CSV resource and, then, do something with this array of errors.
It is extendable, so you can parse your own type of resource/stream, if you have very special needs.
Table of content
- Install
- Usage
- Documentation
- Error management
- Working with database, column optimization
- Guessers : auto detect line ending, delimiter and encoding
- Known issues
- Testing
- Security
- License
Install
Via Composer
Create a composer.json file in your project root:
{ "require": { "voilab/csv": "^5.0.0" } }
$ composer require voilab/csv
Install PHP5 compatible version
This PHP5 version can't parse streams nor iterables.
{ "require": { "voilab/csv": "dev-feature/php5" } }
Usage
Available methods
$parser = new \voilab\csv\Parser($defaultOptions = []); $result = $parser->fromString($str = "A;B\n1;test", $options = []); // or $result = $parser->fromFile($file = '/path/file.csv', $options = []); // or with a raw resource (fopen, fsockopen, php://memory, etc) $result = $parser->fromResource($resource, $options = []); // or with an array or an Iterator interface $result = $parser->fromIterable($array = [['A', 'B'], ['1', 'test']], $options = []); // or with a SPL file object $result = $parser->fromSplFile($object = new \SplFileObject('file.csv'), $options = []); // or with a PSR stream interface (ex. HTTP response message body) $response = $someHttpClient->request('GET', '/'); $result = $parser->fromStream($response->getBody(), $options = []); // or with a custom \voilab\csv\CsvInterface implementation $result => $parser->parse($myCsvInterface, $options = []);
Simple example
$parser = new \voilab\csv\Parser([ 'delimiter' => ';', 'columns' => [ 'A' => function (string $data) { return (int) $data; }, 'B' => function (string $data) { return ucfirst($data); } ] ]); $csv = <<<CSV A; B 4; hello 2; world CSV; $result = $parser->fromString($csv); foreach ($result as $row) { var_dump($row['A']); // int var_dump($row['B']); // string with first capital letter }
Full example
$parser->fromFile('file.csv', [ // fgetcsv 'delimiter' => ',', 'enclosure' => '"', 'escape' => '\\', 'length' => 0, 'autoDetectLn' => null, // resources 'metadata' => [], 'close' => false, // PSR stream 'lineEnding' => "\n", // headers management 'headers' => true, 'strict' => false, 'required' => ['id', 'name'], // big files 'start' => 0, 'size' => 0, 'seek' => 0, 'chunkSize' => 0, // data pre-manipulation 'autotrim' => true, 'onBeforeColumnParse' => function (string $data) { return utf8_encode($data); }, 'guessDelimiter' => new \voilab\csv\GuesserDelimiter(), 'guessLineEnding' => new \voilab\csv\GuesserLineEnding(), 'guessEncoding' => new \voilab\csv\GuesserEncoding(), // data post-manipulation 'onRowParsed' => function (array $row) { $row['other_stuff'] = do_some_stuff($row); return $row; }, 'onChunkParsed' => function (array $rows) { // do whatever you want, return void }, 'onError' => function (\Exception $e, $index) { throw new \Exception($e->getMessage() . ": at line $index"); } // CSV columns definition 'columns' => [ 'A as id' => function (string $data) { return (int) $data; }, 'B as firstname' => function (string $data) { return ucfirst($data); }, 'C as name' => function (string $data) { if (!$data) { throw new \Exception("Name is mandatory and is missing"); } return ucfirst($data); }, // use of Optimizers (see at the end of this doc for more info) 'D as optimized' => new \voilab\csv\Optimizer( function (string $data) { return (int) $data; }, function (array $data) { return some_reduce_function($data); } ) ] ]);
Documentation
Options
These are the options you can provide at constructor level or when calling
from*
methods. Details for fgetcsv
options can be found here:
https://php.net/fgetcsv and https://php.net/str_getcsv
Column function parameters
When defining a function for a column, you have access to these parameters:
$parser->fromFile('file.csv', [ 'columns' => [ // minimal usage 'col1' => function (string $data) { return $data; } ] ]);
Headers auto-sanitization
Note that headers are automatically trimmed and their carriage returns are
removed. Also, all spaces following a space are removed. This is only for the
headers. Cells content are not manipulated, except if autotrim
is true.
" a header " => "a header"
"a header" => "a header"
"a
header " => "a header"
If the column you defined in your code doesn't exist in CSV resource and doesn't appear in
required
array, the$meta
argument will have a flagphantom
set totrue
. This is the way to know if the column exists or not in the CSV resource during parsing.
On before column parse function parameters
Just before any CSV column data is parsed, a standard method is called so you can operate the same way on every rows and columns data. You can use that to manage encoding, for example.
Be aware of type declaration in your columns functions if you want to return other types from here.
$parser->fromFile('file.csv', [ // minimal usage 'onBeforeColumnParse' => function (string $data) : string { return utf8_encode($data); } ]);
On row parsed function parameters
When a row is completed, you can do something with all that data.
$parser->fromFile('file.csv', [ // minmal usage 'onRowParsed' => function (array $rowData) { return $rowData; } ]);
Aliasing columns
You can define aliases for columns to ease data manipulation. Just write as
to activate this functionality, like CSV column name as alias
.
Alias must not itself contain as
string. But in the CSV resource, the
header can have such a string.
Note that if you have
as
in a CSV resource header, you must alias it in the columns definitions. Otherwise, the parser will not find this column.
$str = <<<CSV A; B ; Just as I said 4; hello; hey 2; world; hi CSV; $parser = new \voilab\csv\Parser(); $result = $parser->fromString($str, [ 'delimiter' => ';', 'columns' => [ 'A as id' => function (string $data) { return (int) $data; }, 'B as content' => function (string $data) { return ucfirst($data); }, 'Just as I said as notes' => function (string $data) { return $data; } ] ]); print_r($result); /* prints: Array ( [0] => Array ( [id] => 4 [content] => Hello [notes] => hey ) [1] => Array ( [id] => 9 [content] => World [notes] => hi ) ) */
Required columns
If you have aliased a column, and it is a required column, you must use the
alias inside the required
option.
$result = $parser->fromString($str, [ 'required' => ['id', 'content'], 'columns' => [ 'A as id' => function (string $data) { return (int) $data; }, 'B as content' => function (string $data) { return ucfirst($data); } ] ]);
No header
If you have no header in you CSV resource, you need to define the parser like this.
$str = <<<CSV 4; hello 2; world CSV; $result = $parser->fromString($str, [ 'columns' => [ '0 as id' => function (string $data) { return (int) $data; }, '1 as content' => function (string $data) { return ucfirst($data); } ] ]); print_r($result); /* prints: Array ( [0] => Array ( [id] => 4 [content] => Hello ) [1] => Array ( [id] => 9 [content] => World ) ) */
Shuffling columns when defining them
You can define your columns in any order you want. You don't need to provide them in the order they appear in the CSV. You just have to match your keys with a header in the CSV resource.
Note that the execution order of the columns are aligned with your code. In the example below, the function
A()
is called afterB()
, even if column A appears first in CSV resource.
$str = <<<CSV A; B 4; hello 2; world CSV; $parser = new \voilab\csv\Parser(); $result = $parser->fromString($str, [ 'delimiter' => ';', 'columns' => [ 'B' => function (string $data) { // first call return ucfirst($data); }, 'A' => function (string $data) { // second call return (int) $data; } ] ]); print_r($result); /* prints: Array ( [0] => Array ( [B] => Hello [A] => 4 ) [1] => Array ( [B] => World [A] => 9 ) ) */
Seek in big files
You can use the seek mechanism to accelerate parsing big files.
Yon can specify the start index. But it is not mandatory. It is used in the error managment, to know which line bugs, or in the other methods calls, where [$index] is given.
You are responsible for keeping [seek] and [start] snychronized. If you don't, and you have errors, the indexes would be irrelevant.
$str = <<<CSV A; B 4; hello 2; world ... CSV; $parser = new \voilab\csv\Parser(); $resource = new \voilab\csv\CsvString($str); $result = $parser->parse($resource, [ 'delimiter' => ';', 'size' => 2, 'columns' => [ 'B' => function (string $data) { return ucfirst($data); }, 'A' => function (string $data) { return (int) $data; } ] ]); $lastPos = $resource->tell(); $resource->close(); $resource2 = new \voilab\csv\CsvString($str); $nextResult = $parser->parse($resource2, [ 'delimiter' => ';', 'size' => 2, 'start' => 2, // yon **can** specify the start index. Not mandatory. 'seek' => $lastPos, 'columns' => [ 'B' => function (string $data) { return ucfirst($data); }, 'A' => function (string $data) { return (int) $data; } ] ]);
Close the resource
Using fromString()
and fromFile()
methods, the resource will be closed
automatically. With other from*()
methods, you can close the resource by
giving the 'close' => true
option.
Line endings problems
Just as stated in official documentation, if you have problems with recognition in line endings, you can use the option below to activate auto detect.
$parser->parse($resource, [ 'autoDetectLn' => true ]);
Note that auto detect PHP ini param is not reseted to initial value after the parsing has finished.
When parsing streams (like HTTP response message body), line ending must be specified in the array options.
Error management
You can use the onError
option to collect all errors, so you can give a
message to the user with all errors in the file you found, in one shot.
You can stop the process of a row by checking the $meta
argument. It has a
key type
which can be row
or column
. If it's column
, you can throw the
error and it will call onError
again, but with type row
. Other columns will
be skipped for this row.
If you use an optimizer, you can call an Exception from there too. The key
type
will then have the value optimizer
.
$errors = []; $data = $parser->fromFile('file.csv', [ 'onError' => function (\Exception $e, $index, array $meta, array $options) use (&$errors) { $errors[] = "Line [$index]: " . $e->getMessage(); // do nothing more, so next columns and next lines can be parsed too. // meta types are the following: switch ($meta['type']) { case 'init': case 'column': case 'row': case 'reducer': case 'optimizer': case 'chunk': } }, 'columns' => [ 'email' => function (string $data) { // accept null email but validate it if there's one if ($data && !filter_var($data, FILTER_VALIDATE_EMAIL)) { throw new \Exception("The email [$data] is invalid"); } return $data ?: null; } ] ]); if (count($errors)) { // now print in some ways all the errors found print_r($errors); } else { // everything went well, put data in db on whatever }
Initialization errors
Some errors are thrown before any line is parsed. You have to take this into account.
$data = $parser->fromFile('file.csv', [ 'onError' => function (\Exception $e, $index, array $meta) { if ($meta['type'] === 'init') { // called during initialization. var_dump($meta['key']); // for errors with specific key if ($e->getCode() === \voilab\csv\Exception::HEADERMISSING) { throw new \Exception(sprintf("La colonne [%s] est obligatoire", $meta['key'])); } } throw $e; } ]);
Error and internationalization (i18n)
If you want to translate error messages, you can use the onError
function
with meta['type'] === 'init'
to throw the translated message.
Working with database, column optimization
When parsing large set of data, if one column is, for example, a user ID, it's
a bad idea to call a find($id)
method for each CSV row iteration. It's better
to take all column values, and call for a findByIds($ids)
.
The build-in class Optimizer
allows you to define a column this way. It takes
three arguments. The first is the function needed to parse value from CSV.
The second is a reduce function. It recieves all data of the column, and must
return an indexed array.
For example, if you have 2 rows with values a
and b
, the indexed result of
the reduce function would be Array ( a => something, b => something else )
.
The third argument is a function called when a value is not found in the reduced function.
Parse function
Same as Column function (see above)
Reduce function
Returns an indexed array. If there's no correspondance between CSV column values and the result of the reduce function, you should not return the missing value. For example, if values are [10, 22], they are used in database query to find users by id, and user ID 22 doesn't exist, the result should be
Array ( 10 => User(id=10) )
Absent function
When a value is not found in the reduced result, the default behaviour is to set the value (like there wasn't any reduce function for this row). You can override this by defining the absent function, and do what you want with the value.
If you have defined an error function, it will be called with a type of
optimizer
(check error management above) if you throw an error from here.
Example
$str = <<<CSV A; B 4; updated John 2; updated Sybille CSV; $database = some_database_abstraction(); $data = $parser->fromString($str, [ 'delimiter' => ';', 'columns' => [ 'A as user' => new \voilab\csv\Optimizer( // column function, same as when there's no optimizer function (string $data) { return (int) $data; }, // reduce function that uses the set of datas from the 1st function function (array $data) use ($database) { $query = 'SELECT id, firstname FROM user WHERE id IN(?)'; $users = $database->query($query, array_unique($data)); return array_reduce($users, function ($acc, $user) { $acc[$user->id] = $user; return $acc; }, []); }, // absent function. data is [int] because the first function returns // an [int] function (int $data, int $index) { throw new \Exception("User with id $data at index $index doesn't exist!"); } ), 'B as firstname' => function (string $data) { return $data; } ] ]); print_r($result); /* prints: Array ( [0] => Array ( [user] => User ( id => 4, firstname => John ) [firstname] => updated John ) [1] => Array ( [user] => User ( id => 2, firstname => Sybille ) [firstname] => updated Sybille ) ) */
Chunks
Optimizers are good in certain cases, but sometimes you want to parse your data by chunk, maniuplate it, store it, and do it again with the next chunk. You can achieve this with chunks options:
$str = ''; // a hudge CSV string with tons of rows and two columns $parser->fromString($str, [ 'delimiter' => ';', 'chunkSize' => 500, 'onChunkParsed' => function (array $rows, int $chunkIndex, array $columns, array $options) { // count($rows) = 500 // do something with your parsed rows. This method will be called // as long as there are rows to parse. // This method returns void }, 'onError' => function (\Exception $e, $index, array $meta) { // if ($meta['type] === 'chunk') { do something } }, 'columns' => [ 'A as name' => function (string $data) { return (int) $data; }, 'B as firstname' => function (string $data) { return $data; } ] ]);
If you use optimizers,
$rows
will be the resultset optimized.
You don't need to use the array returned by
fromString
(or alike) because what you did inonChunkParsed
is enough.
Guessers : auto detect line ending, delimiter and encoding
Guessing how CSV data is structured (line ending, delimiter or encoding) is a very hasardous task, with so many use-cases it's impossible to rule them all..
This package still provides a way to guess these elements, but if it doesn't fit your needs, you can easily extend or create a new class and manage your specific use-case.
If you want to implement guessing your own way, please read the code base for each guessing interfaces.
Guessing is useless with some CsvInteface implementations. For example, iterables are ignored, since data is already arranged in cells and rows. Be sure it's useful for you before you use guessing features.
Guess line ending
First thing the parser does is to detect which are the line endings. The
provided implementation tries to detect line endings among \n
, \r
and \r\n
.
$str = 'A;B\r\n4;Hello\r\n;2;World'; $parser->fromString($str, [ 'guessLineEnding' => new \voilab\csv\GuesserLineEnding([ // maximum line length to read, which will be parsed // defaults to: see below 'length' => 1024 * 1024 ]) ]);
Guess delimiter
Then, the parser tries to detect delimiter. In the provided implementation, an exception is thrown if delimiter is not found or if it's too ambiguous.
$str = 'A;B\n4;Hello\n;2;World'; $parser->fromString($str, [ 'guessDelimiter' => new \voilab\csv\GuesserDelimiter([ // delimiters to check. Defaults to: see below 'delimiters' => [',', ';', ':', "\t", '|', ' '], // number of lines to check. Defaults to: see below 'size' => 10, // throws an exception if result is amiguous. Defaults to: see below 'throwAmbiguous' => true, // score to reach for a delimiter. Defaults to: see below 'scoreLimit' => 50 ]) ]);
Guess encoding
For each cell, encoding auto detection is called. The provided implementation tries to find the current cell encoding, and encode it to the other one given in the constructor.
It is also called for the header row. If you want to encode differently
between headers and datas, you can check on $meta['type'] === 'init'
in
your encode
function (check code base).
This guesser is called BEFORE onBeforeColumnParse
$str = 'A;B\n4;Hellö\n;2;Wörld'; $parser->fromString($str, [ 'guessEncoding' => new \voilab\csv\GuesserEncoding([ // encoding in which data to retrieve. Defaults to: see below 'encodingTo' => 'utf-8', // encoding in file. If null, is auto-detected 'from' => null, // available encodings. If null, uses mb_list_encodings 'encodings' => null, // strict mode for mb_detect_encoding. Defaults to: see below 'strict' => false ]) ]);
Known issues
- with PSR streams, carriage returns are not supported in headers and in cells content
- guessing processes are unlikely to fit your specific needs immediately. Before creating an issue or a PR, try to extends the guess classes and make your own specific adaptations
Testing
$ /vendor/bin/phpunit
Security
If you discover any security related issues, please use the issue tracker.
License
The MIT License (MIT). Please see License File for more information.