tinytestJUnit test results

Description

An object of class tinytests2JUnit. Note the plurar. A subclass of tinytest::tinytests() containing extra info recordings that are used in the export to JUnit.

Usage

## S3 method for class 'tinytests2JUnit'
x[i]

Arguments

Details

Following details are recorded when running the tests files and stored as additional attributes to the object:

• fileDurations: named-numeric(n). Names = filename of tests files, value = duration in seconds on how long the test file took to run.

• fileTimestamps: named-character(n). Names = filename of tests files, value = timestamp when the test was invoked.

• fileHostnames: named-character(n). Names = filename of tests files, value = The hostname of the system that ran the tests. (Usefull in combination with clusters).

• disabled: character. A character vector of filenames where no tests were ran. They are flagged as disabled tests.

Convert any will character vector to a single length character vector

Description

Convert any will character vector to a single length character vector

Usage

charVecToSingleLength(x)

Arguments

Value

x a single-length character vector Non-NA

Examples

tinytest2JUnit:::charVecToSingleLength(c("Hello", "World")) # -> "HelloWorld"
tinytest2JUnit:::charVecToSingleLength(c("Hello", NA_character_)) # -> "HelloNA"
tinytest2JUnit:::charVecToSingleLength(character(0L)) # -> ""

Helper function specifying the 'classname' attribute of the testcase tag.

Description

Helper function specifying the 'classname' attribute of the testcase tag. Currently equal to the fileName. The classname is already xml escaped.

Usage

classnameTestcase(tinytest)

Arguments

Value

character(1) being the 'classname'

Helper function generating the body of a failure description tag!

Description

Helper function generating the body of a failure description tag! Attempts to mimic the print behaviour of a tinytest object.

Usage

constructFailureDescription(tinytest)

Arguments

Value

character(1) being the failure tag description body. This string is already propery xml escaped.

Construct JUnit ⁠</testcase>⁠ tag

Description

Construct JUnit ⁠</testcase>⁠ tag based on a single tinytest result.

Usage

constructTestcaseTag(tinytest)

Arguments

Value

XMLtag: with tag-name = tinytest and contains the test result per test.

Construct JUnit ⁠</testsuite>⁠ tag

Description

Construct the ⁠</testsuite>⁠ tag of a tinytest, given all the tinytest results from a single test file.

Usage

constructTestsuiteTag(testsFile, id)

Arguments

Details

In case a tinytest2JUnit is provided following additional info can be reported:

• testsuite duration.

• timestamp when the testsuite was performed.

• hostname where the testsuite was ran.

Value

XMLtag: with tag-name = ⁠</testsuite>⁠ that contains all the test results per test file.

Construct the JUnit ⁠</testsuites>⁠ tag

Description

Convert the tinytests2Junit or tinytests-object containing test across possibly multiple files into a JUnit ⁠</testsuites>⁠ tag. More details are reported to the JUnit if a tinytests2JUnit object compared to the native tinytests object.

Usage

constructTestsuitesTag(testResults)

Arguments

Details

Reference for JUnit XML format: https://llg.cubic.org/docs/junit/

See details runTestDir() which additional info is recorded.

Value

XMLtag: with tag-name = ⁠</testsuites>⁠. This is the root of the JUnit XML document.

Construct a testcase-tag for an error test.

Description

Construct a testcase-tag for an error test.

Usage

errorTestcaseTag(tinytest)

Arguments

Value

a testase XMLtag

Escape xml

Description

Escape the characters &,⁠"⁠,⁠'⁠,<,>

Usage

escapeXml(x)

Arguments

Value

The same character vector x but xml escaped.

See Also

https://stackoverflow.com/a/1091953/10415129

Escape xml text

Description

Escape the characters '<' and & in a character vector meant to be xml-text content.

Usage

escapeXmlText(x)

Arguments

Value

The same character vector x but xml text escaped.

Construct a testcase-tag for a failed test.

Description

Construct a testcase-tag for a failed test.

Usage

failureTestcaseTag(tinytest)

Arguments

Value

a testase XMLtag

Format method for XMLtag class

Description

Format S3 method for the XMLtag-class

Usage

## S3 method for class 'XMLtag'
format(x, level = 0, ...)

Arguments

Details

Note, text content does not get indented or put on a new line, since whites space characters are of relevance.

Value

character(1) vector of the formatted XML tag.

Help function to generate the formatted string for a single stack frame.

Description

Help function to generate the formatted string for a single stack frame.

Usage

formattedFrame(framecall, frameN, hasSrcInfo, dirName, fileName, lineNr)

Arguments

Details

For a given frame in the stack the string is formatted as follows (substitute the arguments between the curly braces) {frameN}| {call[1]} {frameN}| {call[2]} {frameN}| {call[3]} {frameN}| {call[3]} ---> at File={dirName/fileName} Line={line}:

For example for only a single line error: 1: stop("This is a crash") ---> at File=R/my_r_code_file.R Line=234

Currently all call lines are printed for a given stack. The last line with source file info only printed if hasSrcInfo=TRUE. Else it is ommited.

Value

characer(1) the formatted character string containing info of a single frame in the stackstrace

Get formatted stack trace for an uncaught error from a tinytest test file.

Description

getFormattedStacktrace is a helper function that formats stacktrace for uncaught errors from a tinytest run file.

Usage

getFormattedStacktrace()

Details

This function assumes that it directly called from a withCallingHandler error handling function! This fact is then used to remove the calling handler info from the stack such that stack directly starts from where the error was thrown.

The function also removes the calls from the stack that involve executing the test_file. The internals of runTestDir and tinytest are not of intererst. And the highest level of the stack to consider is the top level of the test_file.

Note, this does mean that errors that occur on the top-level of the test file will not have a a stacktrace! For example: "Error: object 'x' not found" where x is attempted to be resolved at the root levels

Value

character(1) a single length character string suitable to be printed to the end-user. In case of no stakctrace (eg the error occured at root level of the script) NA_character_ is returned!

Test if single length character non NA.

Description

Test if single length character non NA.

Usage

isSingleLengthCharNonNA(x)

Arguments

Value

logical(1)

Author(s)

ltuijnder

Helper function to construct the name of a testcase

Description

Helper function to construct the name of a testcase. Note, the charater is already xml escaped.

Usage

nameTestcase(tinytest)

Arguments

Value

character(1) the testcase name to use for this tinytest object.

Construct a testcase-tag for a passed test.

Description

Construct a testcase-tag for a passed test.

Usage

passedTestcaseTag(tinytest)

Arguments

Value

a testase XMLtag

Print method for XMLtag class.

Description

Print method for XMLtag class.

Usage

## S3 method for class 'XMLtag'
print(x, ...)

Arguments

Value

invisibly the string that was printed to stdout.

Run all the test files in a directory

Description

runTestDir() is a drop in replacement for tinytest::run_test_dir() with the key difference that errors thrown from within a test file are caught and get reported with a a stacktrace in the JUnit report. In addition, some extra metrics are recored for the JUnit report, such as: timestamp, test duration, hostname and if tests are disabled (see details for more info).

Usage

runTestDir(
  dir = "inst/tinytest",
  at_home = FALSE,
  pattern = "^test.*\\.[rR]$",
  cluster = NULL,
  lc_collate = getOption("tt.collate", NA),
  ...
)

Arguments

Details

runTestDir() is meant as a CI-friendly alternative to the native tinytest::run_test_dir(). It catches errors that are raised in the tests files and adds them as a "failed" tinytest in the output.

tinytest::run_test_dir() would have let the error bubble up, stop the testing process and not report any failures from other tests. One is then also forced to look into the logs of the CI to see what the error was. The output of runTestDir() in combination with writeJUnit() will present you the error in the JUnit togheter with a stack trace. Next to the test results of the other files that ran without a problem.

If you prefer the behaviour of tinytest::run_test_dir() you can still use it in combination writeJUnit().

Caught errors are returned in the output as as sub-class of tinytest object. This is however considred implemenation detail and can be subject to change.

Note, function arguments explicilty listed in tinytest::run_test_dir() but not here can still still be provided via ...

Value

A tinytests2Junit object to be provided to the writeJUnit() function.

tinytests2JUnit

The returned object is a tinytests2JUnit object (note the plural). This object contains additional info compared to a tinytests object that is used in the JUnit report.

The following additional info will get reported:

• The timestamp per test file on when it got invoked.

• The test duration per test file.

• The system hostname per test file on where it got invoked. This is mainly of interests for different clusters.

• If a test file is disabled. A test file is considered disabled if no tests occur with in the file. It is then assumed that at the top of file some conditional statement made the test file exist early.

See Also

• tinytest::run_test_dir() for how the function is inteded to behave.

• writeJUnit() where it is expected that the output of this function to be provided to.

• testPackage() for an higher-level function to simply test a package.

Examples

# Run tests with `tinytest`
dirWithTests <- system.file("example_tests/multiple_files",package = "tinytest2JUnit")
testresults <- runTestDir(dirWithTests)

writeJUnit(testresults) # Writes content to stdout

Internal wrapper arround tinytest::run_test_file

Description

Internal wrapper arround tinytest::run_test_file() that records the test duration and catches uncaught errors and logs the stacktrace of where the error occured.

Usage

runTestFile(file, ...)

Arguments

Details

The response is a subclass of the tinytests object called: tinytests2Junit object which captures additional info for the reporting to JUnit:

• Duration to run the file.

• Timestamp when the test was run.

• hostname of the computer where it was ran on.

The caught error is turned into a subclass uncaught-error of tinytest. This is implementation detail and only to be understood by constructJUnitTag.

If an error occured it is captured and uncaught-error object (subclass of tinytest) is returned in the tinytests object. This tinytest object represents a "failed" tests that will get reported as an Error in the JUnit. Various aspects of the error are also captured like the the stacktrace.

Value

a tinytests2JUnit object (being a subclass of tinytest object).

Construct a testcase-tag for a side-effect test.

Description

Construct a testcase-tag for a side-effect test.

Usage

sideeffectTestcaseTag(tinytest)

Arguments

Value

a testase XMLtag

XML tag

Description

Create a list object that roughly mimics the behaviour of a simplistic XML tag element. Supported are XML tag-name, tag-attributes and tag-content.

Usage

tag(name, attributes = list(), content = list())

Arguments

Details

If a character vector is in the content it is converted to a single-length character vector. See charVecToSingleLength()

Mixed content eg. a text string and a child xml tag next to each other is syntaxtically allows. In practices it does not occur for XML that is schema formatted with XSD (like JUnit). So for simplicity it is not supported here.

Value

a XMLtag-object.

Test an R package and report the results in JUnit

Description

Run all tests of a package and report the results as JUnit xml. This function can be seen as a drop in replacement for tinytest::test_package() but with a key difference that uncaught errors will be catched and reported JUnit! This function is intended to be used in a test stage of a CI build.

Usage

testPackage(
  pkgname,
  file = stdout(),
  errorOnFailure = TRUE,
  testdir = "tinytest",
  lib.loc = NULL,
  at_home = FALSE,
  ncpu = NULL,
  ...
)

Arguments

Details

testPackage() is meant as a CI-friendly alternative to the native tinytest::test_package(). Next to directly reporting the tests results in a JUnit xml format, it also catches errors that are raised in the tests files and reports them as "error" in the JUnit.

tinytest::test_package() would have let the error bubble up, stop the testing process and not report any failures from other test files. One is then also forced to look into the logs of the CI to see what the error was. testPackage() presents you that error in the JUnit with a stacktrace. Next to all the test results of the other files that ran without a problem.

If you prefer the behaviour from tinytest::test_package(), you can still use it in combination with writeJUnit() if all tests results pass.

Just like tinytest::test_package() an error is raised if at least one failure occured during testing. Obviously catched errors are also seen as failures. This error is raised after the test results have been written away to the file, such that your CI can still pick it up and report the failure. The error raising is done as a convenience to stop the CI from continue if test-failure occured. You can opt-out of this behaviour by setting the errorOnFailure parameter to FALSE. Then a case tinytests2JUnit object is returned (a sub-class of tinytests object containing addition info for the JUnit). Caught errors are also captured in this object as tinytest-objects. They actually have a special sub-class but this is considered an internal implemenation detail.

testPackage() is NOT meant to be called from within your tests/tinytests.R file! Tests invoked by R CMD Check or on CRAN should still make use of tinytest::test_package(). This function is only meant to be called from within a testing step in your CI to report the test results in an JUnit xml format.

Value

If errorOnFailure = FALSE, a tinytests2JUnit object (a subclass of tinytests object that captures more info for export to JUnit). Else, an error is raised if at least on failure occurs. Meant as convenience to automatically stop the CI build.

Side-effects

Side effects are registered as 'passed' tests in the JUnit output and have been given a status "SIDE-EFFECT". The call and diff is also returned in the standard-output of the testcase tag.

They are not considred failures and would thus not stop a pipeline.

tinytests to JUnit

To comply the the JUnit specification the tests results are adapted as follows:

• A single test run tinytests is mapped to a ⁠<testsuites>⁠ tag.

• All tinytest results from a single file are mapped to a single ⁠<testsuite>⁠ tag.

  • The name of the testsuite is equal to the test file name (without the file suffix)

• An individual tinytest object (eg. a single ⁠except_*⁠ exception test) is mapped to a ⁠<testcase>⁠ tag.

  • The name of the testcase is equal to the fileName + Line specification of where the expect statement is performed + the info.

For reference: https://llg.cubic.org/docs/junit/

See Also

runTestDir() and tinytest::test_package().

Examples

tmpFile <- tempfile(fileext = ".xml")
testPackage("tinytest", file = tmpFile, verbose = 0)

Write the results of a tinytests-object into JUnit xml report.

Description

Write the tinytests-object to a JUnit XML reporting file. If a tinytests2JUnit is provided (returned by runTestDir()) more info will get reported.

Usage

writeJUnit(tinytests, file = stdout(), overwrite = TRUE)

Arguments

Value

invisible(TRUE) Might get another use in the future.

Errors

In case of overwrite = FALSE and the file already exists an error is thrown.

Side-effects

Side effects are registered as 'passed' tests in the JUnit output and have been given a status "SIDE-EFFECT". The call and diff is also returned in the standard-output of the testcase tag.

They are not considred failures and would thus not stop a pipeline.

tinytests to JUnit

To comply the the JUnit specification the tests results are adapted as follows:

• A single test run tinytests is mapped to a ⁠<testsuites>⁠ tag.

• All tinytest results from a single file are mapped to a single ⁠<testsuite>⁠ tag.

  • The name of the testsuite is equal to the test file name (without the file suffix)

• An individual tinytest object (eg. a single ⁠except_*⁠ exception test) is mapped to a ⁠<testcase>⁠ tag.

  • The name of the testcase is equal to the fileName + Line specification of where the expect statement is performed + the info.

For reference: https://llg.cubic.org/docs/junit/

See Also

The JUnit XML report format: https://llg.cubic.org/docs/junit/

Examples

# Run tests with `tinytest`
dirWithTests <- system.file("example_tests/multiple_files",package = "tinytest2JUnit")
testresults <- runTestDir(dirWithTests)

writeJUnit(testresults) # Writes content to stdout

tmpFile <- tempfile(fileext = ".xml")
writeJUnit(tinytests = testresults, file = tmpFile)