scrutiny - scrutiny 0.5.0

The package is now released under the MIT license.

Breaking changes

• The ratio column in the output of grim_map() and grim_map_seq() was replaced by a probability column. This means:

  • Numerically, the only difference is that probability is zero whenever ratio was negative.

  • Conceptually, it is much easier to interpret: it is the probability that a reported mean or percentage of integer data that has a specific number of decimal places but is otherwise random is GRIM-inconsistent with the reported sample size.

    For example, probability is 0.6 for a mean of 1.23 and a sample size of 40. The same is true for any other mean with two decimal places. Thus, a randomly chosen mean with two decimal places, ostensibly derived from integer data, has a 0.6 probability of being GRIM-inconsistent with the reported sample size.

• In the functions around grim_ratio(), the x argument must now be a string. This is consistent with grim_map(), unround(), etc.; and it prevents erroneous results that could previously occur by omitting trailing zeros.

• The GRIMMER implementation was debugged, so that grimmer_map() etc. may now yield different results in a few cases. In particular, the items argument now works correctly, thanks to Aurélien Allard and Lukas Wallrich (#58).

• is_seq_dispersed() now correctly returns FALSE if different numbers of missing values at the start and end of x mean that x cannot be dispersed around from.

New features

• The probability column (see above) is created by a new function, grim_probability().

Lifecycle updates

Deprecated

• As a consequence of the above, the show_prob argument of grim_map() is now deprecated and will be removed in a future version. It no longer has any effect.

• grim_ratio_upper() is deprecated and will be removed in a future version. It no longer seems very interesting (and likely never was), especially now that the GRIM ratio in general has taken a backseat.

• All 15 (!) functions around is_subset_of() are deprecated and will be removed in a future version. In truth, they were always poorly written and widely out of scope for scrutiny.

Removed

All of these had been deprecated since scrutiny 0.3.0:

• audit_list() was removed.

• The sep argument in restore_zeros() and restore_zeros_df() was removed.

• The numeric_only argument in duplicate_count() and duplicate_detect() was removed.

• The na.rm argument in duplicate_count_colpair() was removed.

- R
Published by lhdjung almost 2 years ago

scrutiny - scrutiny 0.4.0

This version brings major performance improvements. Furthermore:

Bugfixes

• Fixed a bug in audit_seq(): If the dispersion argument in the preceding call to a function like grim_map_seq() was specified as something other than a linearly increasing sequence, the "diff_*" columns in the data frames returned by audit_seq() may have contained incorrect values.
• Similarly, audit_seq() and reverse_map_seq() used to reconstruct the reported values incorrectly if the dispersion default was overridden as described above. At least for now, the issue is handled by throwing an error if these functions operate on data frames that are the result of specifying dispersion as something other than a linearly increasing sequence.
• Fixed a bug that incorrectly threw an error in grim_map_seq(), other functions made by function_map_seq(), as well as seq_disperse() and seq_disperse_df() if an input value was so close to out_min or out_max that the output sequence would be shorter than implied by dispersion / .dispersion , and if track_var_change / .track_var_change (see below) was TRUE. Again, note that the bug only occurred if an error was thrown.

New features

• A new vignette lists the options for specifying the rounding argument that many scrutiny functions have: vignette("rounding-options").
• Another new vignette shows the minimal steps to implement a consistency test using scrutiny: vignette("consistency-tests-simple").
• The output of grim_map_seq(), grimmer_map_seq(), debit_map_seq() and any other function made by function_map_seq() now has a diff_var column that tracks the difference between the dispersed variable (see the var column) and the reported value. Following the diff_* columns in the output of audit_seq(), this is the number of dispersion steps, not the actual numeric difference.
• The same diff_* columns are now integer, not double.
• function_map(), function_map_seq(), and function_map_total_n() have a new .name_key_result argument that controls the name of the key result column in the output of the factory-made function. This is "consistency" by default, but other names will fit better for other kinds of tests. (The results of these tests must still be logical values.)

Minor changes

• In duplicate_count(), the count column in the output tibble was renamed to frequency. This makes for a more streamlined frequency table and removes an ambiguity with duplicate_count_colpair(), where the count output column means something different.
• In seq_disperse() and seq_disperse_df(), the track_var_change / .track_var_change argument was renamed to track_diff_var / .track_diff_var. The arguments with the old names are still present for now but will be removed in a future version. Also, the unit of these values is now dispersion steps, for consistency with grim_map_seq() etc. as well as audit_seq().
• grim_total(), grim_ratio(), and grim_ratio_upper() now require x to have length 1.
• The docs now link to functions when opened in RStudio, not just on the website.
• Accordingly, the output of write_doc_factory_map_conventions() now renders links. The function also has a new scrutiny_prefix argument for use in another package.
• The "Infrastructure" article was renamed to "Developer tools"; vignette("devtools").
• Some dependencies that used to be suggested are now imported.

- R
Published by lhdjung over 2 years ago

scrutiny - scrutiny 0.3.0

Duplicate analysis overhaul

The duplicate_*() functions now present their output better and have overall been streamlined. Read more at vignette("duplicates").

• A new function, duplicate_tally(), marks each observation with its overall frequency. It is similar to duplicate_detect() but more informative.

• In duplicate_count():

  • All values are now treated like character strings, so all can be checked. The numeric_only argument is deprecated and should no longer be used.
  • The output tibble has two new columns, locations and locations_n. These hold the names of all input columns in which a value appears and the number of these columns. Details are controlled by the new locations_type argument.
  • New ignore argument for specifying one or more values that will not be checked for duplicates.

• In duplicate_count_colpair():

  • New total_x and total_y columns in the output show how many non-missing values were checked for duplicates.
  • New ignore argument as in duplicate_count().
  • The na.rm argument is deprecated. It wasn't very useful because missing values are never checked for duplicates.

• duplicate_detect() is superseded. It is less informative than duplicate_count() and, in particular, duplicate_tally(). Still, it shares in the overhaul:

  • As in duplicate_count(), all values are now treated like character strings, so all can be checked. The numeric_only argument is deprecated and should no longer be used.
  • The duplicate status of missing values is now shown as NA.
  • New ignore argument as in duplicate_count().

Bugfixes

• Fixed a numeric precision bug in round_up_from() and round_down_from() that occurred when rounding numbers greater than circa 2100 with a part to be truncated that was equal to 5 on that decimal level (thanks to \@kaz462, #43). These functions are called within round_up() and round_down(), and indirectly by all consistency-testing functions.

• Fixed a bug in audit_seq() that displayed one "hit" found by varying a given reported value if there were no such hits. The other columns were not affected.

• Fixed a bug in function_map() that displayed the wrong calling function's name in case of an error.

Minor improvements

• Documentation for grim_map_seq() and all other functions made by function_map(), function_map_seq(), or function_map_total_n() now displays meaningful defaults. Printing the factory-made functions is more meaningful, as well. Internally, they now work with rlang::new_function(), which allows for flexible metaprogramming.

• The experimental function audit_list() is deprecated. Just use audit() instead. audit_seq() and audit_total_n() are now documented separately from audit() and audit_list().

• The lifecycle package is now imported and used in formal deprecations such as that of sep in the restore_*() functions. The janitor package is no longer suggested.

• Adjusted to new CRAN requirements for packageVersion() usage.

• Some performance improvements.

- R
Published by lhdjung almost 3 years ago

scrutiny -

scrutiny 0.2.3

Some new features and bugfixes:

• New audit() methods for the output of audit_seq() and audit_total_n().

• New duplicate_count_colpair() function that checks each combination of columns in a data frame for duplicates.

• New restore_zeros_df() function to easily restore trailing zeros in all numeric-like columns in a data frame.

• New seq_length() function to extend or shorten linear sequences.

• Bugfixes in the is_seq_*() functions.

• Some possible corner case issues in split_by_parens() are now prevented.

• Internal changes for compatibility with purrr 1.0.0 and tidyselect 1.2.0.

- R
Published by lhdjung over 3 years ago

scrutiny - Release of scrutiny 0.2.2

scrutiny 0.2.2.9000 (development version)

"Releasing" the development version right now in order for the website (https://lhdjung.github.io/scrutiny/) to flag a bug in the GRIMMER implementation, concerning the items argument used for composed scales.

scrutiny 0.2.2

This is a patch for CRAN compliance.

• The package now requires R version >= 3.4.0 and rlang version >= 1.0.2.

• Subtle changes to split_by_parens() that users generally won't notice.

• Minor shifts in the documentation (e.g., vignette("consistency-tests") now has instructions on exporting factory-made functions.).

What's Changed

• Devel by @lhdjung in https://github.com/lhdjung/scrutiny/pull/3
• Devel by @lhdjung in https://github.com/lhdjung/scrutiny/pull/4
• remove SPRITE reference: CRAN checks by @lhdjung in https://github.com/lhdjung/scrutiny/pull/5
• Devel by @lhdjung in https://github.com/lhdjung/scrutiny/pull/6
• updated Authors@R field in DESCRIPTION by @lhdjung in https://github.com/lhdjung/scrutiny/pull/7
• correct renamed function; also cut repeat word by @lhdjung in https://github.com/lhdjung/scrutiny/pull/8
• Devel by @lhdjung in https://github.com/lhdjung/scrutiny/pull/9
• add "Resubmission" header in CRAN comments by @lhdjung in https://github.com/lhdjung/scrutiny/pull/10
• add trailing slash to second URL in DESCRIPTION by @lhdjung in https://github.com/lhdjung/scrutiny/pull/11
• Devel by @lhdjung in https://github.com/lhdjung/scrutiny/pull/12
• Devel by @lhdjung in https://github.com/lhdjung/scrutiny/pull/13
• Devel by @lhdjung in https://github.com/lhdjung/scrutiny/pull/14

New Contributors

• @lhdjung made their first contribution in https://github.com/lhdjung/scrutiny/pull/3

Full Changelog: https://github.com/lhdjung/scrutiny/commits/v0.2.2

- R
Published by lhdjung almost 4 years ago