meetmatt / open-api-coverage
OpenAPI specification coverage
dev-master
2024-10-21 22:11 UTC
Requires
- php: ^8.2
- ext-json: *
- cebe/php-openapi: ^1.7
- psr/http-message: ^2.0
- symfony/console: ^7.1
Requires (Dev)
- codeception/codeception: ^5.1
- codeception/module-asserts: ^3.0
- guzzlehttp/psr7: ^2.7
This package is auto-updated.
Last update: 2024-11-21 22:21:39 UTC
README
Purpose
This package can be used to report the documented and record the executed parts of an Open API specification under test: which paths and operations were executed by the test, which parameters were passed, which content types were sent, which response status code classes, codes, body properties and content types were returned and asserted.
Glossary
- Documented / undocumented: present in the specification / not present.
- Passed / not passed: sent in the request / not sent.
- Obtained / not obtained: received in the response / not received.
- Tested / not tested: obtained response was asserted against the expected value / not asserted or the given assertion failed.
- TCL: Test Coverage Level (per suite, per specification).
- Input criteria: related to the API requests.
- Output criteria: related to the API responses.
Criteria
Input criteria
Types of input criteria:
- Input parameters (optional or mandatory).
- Input parameter values (enums, booleans).
- Content-types (only applicable for operations with request body).
- Operation flows (not implemented, lacks formal definitions).
Request coverage:
- Documented, passed - fully covered.
- Documented, not passed - lacks API call.
- Undocumented, passed - lacks documentation.
- (Nonsense) Undocumented, not passed.
Output criteria
Types of output criteria:
- Status code classes: 200 vs 400/500, should be explicitly defined.
- Status codes.
- Response body properties.
- Content-types.
Response coverage:
- Documented, obtained, tested - fully covered.
- Documented, not obtained, testing not applicable - lacks API call and tests.
- Documented, obtained, not tested - lacks tests.
- Undocumented, obtained, tested - lacks documentation.
- Undocumented, obtained, not tested - lacks documentation and tests.
- (Nonsense) Undocumented, not obtained, testing not relevant.
Test Coverage Levels (TCL)
Object Model
Coverage
Coverage: Spec: Specification Input: InputCriteria Output: OutputCriteria
Specification
Specification: ID: string Paths: URL -> Path URL: string Operations: HTTP Method -> Operation HTTP Method: string PathParameters: Name -> Values Name: string Values: array QueryParameters: Name -> Values Name: string Values: array RequestBodies: Content-type -> RequestBody Content-type: string Properties: Path -> Values Path: string Values: array Responses: HTTP Status Code -> Response HTTP Status Code: string ResponseBodies: Content-type -> ResponseBody Content-type: string Properties: Path -> Values Path: string Values: array
How the coverage works
- Parse specification.
- On each HTTP call (REST Module) find the path and operation in the spec, mark it as documented and passed.
- Dig deeper to the request parameters: infer path parameters from the URI path, by converting the path definition to regex according to the path parameter, mark the parameter as passed.
- If there's a path, but no operation documented - then mark the path as called, and log an undocumented operation.
- If there's no path, then log an undocumented path (plus operation, parameters (infer path parameters), query string parameters, request body contents).
- Dig deeper to the request parameters: query parameters.
References
License
OpenAPI specification coverage
is open-sourced software licensed under the MIT License.