Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of RJafroc

RJafroc - RJafroc version 2.1.2 is on CRAN!

The need for this release was to fix errors in RJafroc resulting from a recent update of ggplot2.

It also standardizes some the RJafroc functions so as to match my FROC book ; for example, * RSM_FPF() * RSM_TPF() * RSM_NLF() * RSM_LLF() * RSM_wLLF()

- R
Published by dpc10ster over 3 years ago

RJafroc - Version 2.1.1 on CRAN

CRAN resubmission 8/12/22

  • Version 2.1.1, accepted 8/12/22
  • On cran211 branch.
  • Eliminates two html rendering notes occurring on 2.1.0.
  • Corrected 2 URL formatting errors, in DESCRIPTION and RJafroc-package.Rd.

Description: Analyzing the performance of artificial intelligence

(AI) systems/algorithms characterized by a 'search-and-report' strategy. Historically observer performance has dealt with measuring radiologists' performances in search tasks, e.g., searching for lesions in medical images and reporting them, but the implicit location information has been ignored. The implemented methods apply to analyzing the absolute and relative performances of AI systems, comparing AI performance to a group of human readers or optimizing the reporting threshold of an AI system. In addition to performing historical receiver operating receiver operating characteristic (ROC) analysis (localization information ignored), the software also performs free-response receiver operating characteristic (FROC) analysis, where lesion localization information is used. A book using the software has been published: Chakraborty DP: Observer Performance Methods for Diagnostic Imaging - Foundations, Modeling, and Applications with R-Based Examples, Taylor-Francis LLC; 2017: https://www.routledge.com/Observer-Performance-Methods-for-Diagnostic-Imaging-Foundations-Modeling/Chakraborty/p/book/9781482214840. Online updates to this book, which use the software, are at https://dpc10ster.github.io/RJafrocQuickStart/, https://dpc10ster.github.io/RJafrocRocBook/ and at https://dpc10ster.github.io/RJafrocFrocBook/. Supported data collection paradigms are the ROC, FROC and the location ROC (LROC). ROC data consists of single ratings per images, where a rating is the perceived confidence level that the image is that of a diseased patient. An ROC curve is a plot of true positive fraction vs. false positive fraction. FROC data consists of a variable number (zero or more) of mark-rating pairs per image, where a mark is the location of a reported suspicious region and the rating is the confidence level that it is a real lesion. LROC data consists of a rating and a location of the most suspicious region, for every image. Four models of observer performance, and curve-fitting software, are implemented: the binormal model (BM), the contaminated binormal model (CBM), the correlated contaminated binormal model (CORCBM), and the radiological search model (RSM). Unlike the binormal model, CBM, CORCBM and RSM predict 'proper' ROC curves that do not inappropriately cross the chance diagonal. Additionally, RSM parameters are related to search performance (not measured in conventional ROC analysis) and classification performance. Search performance refers to finding lesions, i.e., true positives, while simultaneously not finding false positive locations. Classification performance measures the ability to distinguish between true and false positive locations. Knowing these separate performances allows principled optimization of reader or AI system performance. This package supersedes Windows JAFROC (jackknife alternative FROC) software V4.2.1, https://github.com/dpc10ster/WindowsJafroc. Package functions are organized as follows. Data file related function names are preceded by 'Df', curve fitting functions by 'Fit', included data sets by 'dataset', plotting functions by 'Plot', significance testing functions by 'St', sample size related functions by 'Ss', data simulation functions by 'Simulate' and utility functions by 'Util'. Implemented are figures of merit (FOMs) for quantifying performance and functions for visualizing empirical or fitted operating characteristics: e.g., ROC, FROC, alternative FROC (AFROC) and weighted AFROC (wAFROC) curves. For fully crossed study designs significance testing of reader-averaged FOM differences between modalities is implemented via either Dorfman-Berbaum-Metz or the Obuchowski-Rockette methods. Also implemented is single treatment analysis, which allows comparison of performance of a group of radiologists to a specified value, or comparison of AI to a group of radiologists interpreting the same cases. Crossed-modality analysis is implemented wherein there are two crossed treatment factors and the aim is to determined performance in each treatment factor averaged over all levels of the second factor. Sample size estimation tools are provided for ROC and FROC studies; these use estimates of the relevant variances from a pilot study to predict required numbers of readers and cases in a pivotal study to achieve the desired power. Utility and data file manipulation functions allow data to be read in any of the currently used input formats, including Excel, and the results of the analysis can be viewed in text or Excel output files. The methods are illustrated with several included datasets from the author's collaborations. This update includes improvements to the code, some as a result of user-reported bugs and new feature requests, and others discovered during ongoing testing and code simplification.

- R
Published by dpc10ster almost 4 years ago

RJafroc - Version 2.1.0 on CRAN

CRAN accepted 7/24/22

  • Version 2.1.0
  • On cran210 branch.
  • Steps to reduce file size to less than 5 Mb:
    • Removed RoiData.xlsx.
  • Otherwise identical to developer and master as of 07/7/22.

Extensive changes to handling of lesDist and relWeights 7/19/22

  • Removed unnecessary dimension on lesDist, it is now always 1D
  • Affected files are UtilLesDistrVector.R and UtilLesionWeightsMatrix.R
  • lesDistr and relWeight must have same lengths and sum to unity
  • relWeight = 0 imposes equal weights
  • can remove a lesion by setting the corresponding lesDist entry to zero
  • need further testing on above capability

added Ch19Vig1FrocSampleSize.Rmd 6/7/22

  • Added added Ch19Vig2FrocSampleSize.Rmd and Ch19Vig2FrocSampleSize.Rmd
  • These vignettes got accidentally removed, perhaps because it was failing tests
  • R CMD CHK works
  • Found lots of instances of four backticks, instead of 3; this is probably why some vignettes were not loading on website

created DfWriteExcelDataFile 3/15/22

  • needed to write Nico simulated FROC data to JAFROC format excel file
  • removed JAFROC format writes capability from DfSaveDataFile; it is now in DfWriteExcelDataFile

fixed wAFROC1_dpc 3/11/22

  • needed to divide final value by (K * K2)
  • cpp code unaffected; this only affected R version used to debug
  • checked R code vs cpp; see JT_R_Py_Foms.xlsx in PyJafrocscraps directory

added example to ChisqrGoodnessOfFit code 3/8/22

argument of St functions 1/24/22

  • analysisOption must be DBM or OR
  • Not "ORH"

Clarified weights matrix 1/7/22

  • See test-RSM-formulae.R
  • Search in code for rsm-pred-wafroc-curve: rsmFormulae.R and UtilAnalyticalAucsRSM.R
  • See RJafrocFrocBook, search for rsm-pred-wafroc-curve
  • Push to developer and master

TBDIF 12-26-21

  • TBDIF: To be done in future
  • Created much confusion in RJafrocFrocBook, chapter on 3-fits
  • RsmFormulae.R: This file is a mess.
  • Remove AUCs in PlotRsmOperatingCharacteristics? - these are done in UtilAnalyticalAucsRsm
  • Add to tests?
  • Replace error function with Phi functions?
  • Remove redundant column in weights matrix
  • change lambdaP to lambda and lambda to lambdaI and corresponding changes in book

Changed RSM_yROC to accept physical parameters 12-26-21

  • So as to be consistent with RSM_xROC
  • Created much confusion in RJafrocFrocBook, chapter on 3-fits
  • Added checks for valid RSM parameters in RsmFormulae.R .
  • Changed all such functions in RsmFormulae.R to accept physical parameters.

Fixed PlotRsmOperatingCharacteristics not working for zeta1 = -Inf

  • Global search string "bug fix 12/7/21" to locate all changes.
  • Starting value of for-loop cannot be -Inf; detect it and set to -3

Fixed PlotRsmOperatingCharacteristics returning correct plots but incorrect AUCs

  • Global search string "bug fix 11/24/21" to locate all changes.
  • Basic issue was that I was using zeta1 = -20 instead of the supplied value.

Fixed Issue 73 and deprecated the MRMC file format 10-28-21 - 10-29-21

  • Global search string "T1-RRRC for ROC data #73"" to locate all changes.
  • Basic issue was missing truthTableStr in any file *.imrc when read by DfReadDataFile.
  • See under tests: StSignificanceTestingCadVsRad: Issue T1-RRRC for ROC data #73 for recreation of this issue.
  • Updated R and RStudio.
  • File DfExtracDataset.R was also affected: the change fixes an error that did not get caught before.
  • Updated documentation and links in StSignificanceTestingCadVsRad.R.
  • Note that For non-JAFROC data file formats, the readerID and modalityID fields must be unique integers, as indicated in documentation of DfReadDataFile().
  • Giving thought to removing support for all non-JAFROC formatted files; otherwise I need to maintain support for four file extensions (*.lrc, *.txt, *.csv and *.imrmc) for the simplest data structure (one rating for each modality-reader-case). This is unnecessarily complicating the code. Final resolution: I will support only *.imrmc. Other formats can still be read by DfReadDataFile() and then saved to a JAFROC format file for analysis within the RJafroc package.

Added ability to read Excel format LROC datasets 6/11/21 - 6/14/21

  • Extended DfReadDataFile to accommodate LROC data; added flag lrocForcedMark
  • Must use newExcelFileFormat = T for this capability
  • Added toy LROC files: see inst/extdata/toyFiles/LROC/lroc*.xlsx
  • See ReadJAFROCNewFormat.R, just before final return, for added code
  • Added tests in test-DfReadDataFile().

Corrected sample size vignettes 4/12/21 and 4/14/21

  • Ch19Vig1FrocSampleSize.Rmd and Ch19Vig2FrocSampleSize.Rmd
  • Fixed SsFrocNhRsmModel.R to not return lesion distribution and weights
  • Fixed vignettes that were using the old structure returned by sig. testing function
  • Fixed 2 FROC SS vignettes; fixed SsFrocNhRsmModel.R to do binning internal to the function
  • Added a test for SsFrocNhRsmModel().
  • Updated Rcpp to 1.0.6. NOTE: version 1.0.6.6 created horrendous errors - R aborts.

Intrinsic vs. physical RSM parameters 4/2/21

  • All C++ functions take physical parameters
  • Rest take intrinsic parameters (2 exceptions, like RSM_xROC and RSM_pdfN)
  • Cleanup:
    • PlotRsmOperatingCharacteristics.R,
    • UtilAnallyticalAucsRSM.R,
    • rsmFormulae.R
    • affected related test files: test-RSM-formulae.R and test-model-aucs.R
    • Used goodValues to check that nothing has changed

Moved to RJafrocBook 1/3/21

  • Vignette Ch10Vig1QuickStart
  • Vignette Ch10Vig2QuickStart
  • Function Compare3ProperRocFits.R
  • Associated files in inst: MRMCRuns and ANALYZED

Added functions RSMpdfN and RSMpdfD

  • Needed for Swets predictions in book; but of general utility.
  • Other new functions added of type RSM_*()
  • Need to vectorize all Cpp functions; no need to carry both scalar and vector types.
  • Add tests for new functions RSM_*()

- R
Published by dpc10ster almost 4 years ago

RJafroc - v2.0.1 released

Package name change: Artificial Intelligence Systems and Observer Performance

Reason for submission

  • This is an update to CRAN version (v1.3.2) which installed with no errors, warnings, or notes (2020-03-06) on all platforms. The package is still passing all checks on all platforms (as of 2020-12-10).
  • This update (v2.0.1) includes includes many improvements to the code, some as a result of user-reported bugs and new feature requests, and others discovered during ongoing testing and code simplification since the last successful submission.

DESCRIPTION

Analyzing the performance of artificial intelligence (AI) systems/algorithms characterized by a "search-and-report" strategy. While historically observer performance has dealt with measuring radiologists' performance in search tasks – i.e., searching for lesions in medical images and reporting them - the software described here applies equally to any task involving searching for and reporting arbitrary targets in images. The package can be used to analyze the performance of AI systems, compare AI performance to a group of human readers or optimize the reporting threshold of an AI system. In addition to performing conventional receiver operating characteristic (ROC) analysis (localization information ignored), the software also performs free-response receiver operating characteristic (FROC) analysis, where lesion localization information is integral to the analyzed data. A book using the software has been published: Chakraborty DP: Observer Performance Methods for Diagnostic Imaging - Foundations, Modeling, and Applications with R-Based Examples, Taylor-Francis LLC; 2017. An online update of this book is at https://dpc10ster.github.io/RJafrocBook/. Illustrations of the software (vignettes) are at https://dpc10ster.github.io/RJafroc/. Supported data collection paradigms are the ROC, FROC and the location ROC (LROC). ROC data consists of single ratings per images, where a rating is the perceived confidence level that the image is that of a diseased patient. An ROC curve is a plot of true positive fraction vs. false positive fraction. FROC data consists of a variable number (zero or more) of mark-rating pairs per image, where a mark is the location of a reported suspicious region and the rating is the confidence level that it is a real lesion. LROC data consists of a rating and a location of the most suspicious region, for every image. Four models of observer performance, and curve-fitting software, are implemented: the binormal model (BM), the contaminated binormal model (CBM), the correlated contaminated binormal model (CORCBM), and the radiological search model (RSM). Unlike the binormal model, CBM, CORCBM and RSM predict "proper" ROC curves that do not inappropriately cross the chance diagonal. Additionally, RSM parameters are related to search performance (not measured in conventional ROC analysis) and classification performance. Search performance refers to finding lesions, i.e., true positives, while simultaneously not finding false positive locations. Classification performance measures the ability to distinguish between true and false positive locations. Knowing these separate performances allows principled optimization of reader or AI system performance. RJafroc supersedes Windows JAFROC (jackknife alternative FROC) software V4.2.1, https://github.com/dpc10ster/WindowsJafroc. Package functions are organized as follows. Data file related function names are preceded by "Df", curve fitting functions by "Fit", included data sets by "dataset", plotting functions by "Plot", significance testing functions by "St", sample size related functions by "Ss", data simulation functions by "Simulate" and utility functions by "Util". Implemented are figures of merit (FOMs) for quantifying performance and functions for visualizing empirical or fitted operating characteristics: e.g., ROC, FROC, alternative FROC (AFROC) and weighted AFROC (wAFROC) curves. For fully crossed study designs significance testing of reader-averaged FOM differences between modalities is implemented via either Dorfman-Berbaum-Metz or the Obuchowski-Rockette methods. Also implemented is single treatment analysis, which allows comparison of performance of a group of radiologists to a specified value, or comparison of AI to a group of radiologists interpreting the same cases. Crossed-modality analysis is implemented wherein there are two crossed treatment factors and the aim is to determined performance in each treatment factor averaged over all levels of the second factor. Sample size estimation tools are provided for ROC and FROC studies; these use estimates of the relevant variances from a pilot study to predict required numbers of readers and cases in a pivotal study to achieve the desired power. Utility and data file manipulation functions allow data to be read in any of the currently used input formats, including Excel, and the results of the analysis can be viewed in text or Excel output files. The methods are illustrated with several included datasets from the author's collaborations. This version corrects bugs, simplifies usage of the software and updates the dataset structure. All changes are noted in NEWS.

- R
Published by dpc10ster over 5 years ago

RJafroc - RJafroc V1.3.2 published to CRAN

Reason for revised submission

  • This is an update to CRAN version 1.3.1 which installed with no errors, warnings, or notes (2020-01-13) on all platforms.
  • Recently it was brought to my attention (by Kurt Hornik Kurt.Hornik@r-project.org) that the package was generating errors on R-devel (>r77865) running on several Linux platforms.
  • The errors were identified as likely due to the new default behavior for option stringsAsFactors = FALSE in functions data.frame() and read.table().
  • This update corrects those errors.

- R
Published by dpc10ster about 6 years ago

RJafroc - RJafroc Version 1.3.1

See NEWS.md

- R
Published by dpc10ster over 6 years ago

RJafroc - RJafroc version 1.2.0

Finally! Fighting both CRAN and GitHub is not easy at my age (71). Thanks to Dr. Peter Phillips!! Happy computing.

- R
Published by dpc10ster almost 7 years ago