selenider

Concise, Lazy and Reliable Wrapper for 'chromote' and 'selenium'

https://github.com/ashbythorpe/selenider

Keywords

r web-scraping

Last synced: 10 months ago · JSON representation

Repository

Concise, Lazy and Reliable Wrapper for 'chromote' and 'selenium'

Basic Info

Host: GitHub
Owner: ashbythorpe
License: other
Language: R
Default Branch: main
Homepage: https://ashbythorpe.github.io/selenider/
Size: 9.79 MB

Statistics

Stars: 42
Watchers: 1
Forks: 1
Open Issues: 5
Releases: 5

Topics

r web-scraping

Created about 3 years ago · Last pushed about 1 year ago

Metadata Files

Readme Changelog License Codemeta

README.Rmd

---
output: github_document
---



```{r, include = FALSE}
available <- selenider::selenider_available()
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%",
  eval = available
)
```

```{r, eval = !available, include = FALSE}
message("Selenider is not available")
```

# selenider


[![R-CMD-check](https://github.com/ashbythorpe/selenider/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/ashbythorpe/selenider/actions/workflows/R-CMD-check.yaml)
[![Codecov test coverage](https://codecov.io/gh/ashbythorpe/selenider/graph/badge.svg)](https://app.codecov.io/gh/ashbythorpe/selenider)
[![CRAN status](https://www.r-pkg.org/badges/version/selenider)](https://CRAN.R-project.org/package=selenider)


Traditionally, automating a web browser is often unreliable, especially when using R.
Programmers are forced to write verbose code, utilising inconsistent workarounds (such as using
`Sys.sleep()` to wait for something to happen).

selenider aims to make web testing and scraping in R much simpler, providing a wrapper for either [chromote](https://rstudio.github.io/chromote/)
or [selenium](https://ashbythorpe.github.io/selenium-r/). It is inspired by Java's [Selenide](https://selenide.org/) and Python's [Selene](https://yashaka.github.io/selene/).

Code reliability and reproducibility are essential when writing R code. selenider provides features to
make your scripts work every time they are run, without any extra code:

* Lazy elements: selenider will only try to find an element on the page when it is absolutely necessary.
  Your definitions of HTML elements are separated from their existence on the page, only allowing
  the two to converge when absolutely necessary. In selenider, HTML elements are stored as the directions
  to the element on the page, rather than the element itself. This is much more reliable than the
  alternative since the webpage can constantly change, resulting in elements becoming invalid between their
  creation and use (e.g. the dreaded `StaleElementReferenceException` in Selenium).
* Automatic waiting: selenider will automatically wait for your code to work (e.g. waiting for an input to
  exist and be clickable before actually clicking it), allowing you to write scripts as if your website
  always responds instantly to your interactions.

selenider's other main focus is its API. Its design choices result in concise yet expressive
code that is easy to read and easy to write:

* A global session object results in shorter, more declarative code. It also allows the session to be
  created at the beginning of your script or test, and closed at the end.
* All functions are designed for use with the pipe operator (`|>` or `%>%`); elements can be selected,
  tested and operated on in a single pipeline.
* `elem_expect()` is a powerful way to specify test expectations, with a simple but extensible syntax
  and informative error messages.
* selenider is compatible with automated testing frameworks like [testthat](https://testthat.r-lib.org) and [shinytest2](https://rstudio.github.io/shinytest2/).

## Installation

``` r
# Install selenider from CRAN
install.packages("selenider")

# Or the development version from Github
# install.packages("remotes")
remotes::install_github("ashbythorpe/selenider")
```

Additionally, you must install [chromote](https://rstudio.github.io/chromote/)
or [selenium](https://ashbythorpe.github.io/selenium-r/).
We recommend chromote, as it is quicker and easier to get up and running.
``` r
# Either:
install.packages("chromote")

# Or:
install.packages("selenium")
```

If you are using selenium, you must also have [Java](https://www.oracle.com/java/technologies/downloads/) installed.

Finally, you must have a web browser installed. For chromote, [Google Chrome](https://www.google.com/chrome/) is
required. For selenium, any browser can be used, but [Firefox](https://www.mozilla.org/firefox/new/) is recommended.

## Usage

```{r}
library(selenider)
```

The following code navigates to the [R project website](https://www.r-project.org/), finds the link to
the CRAN mirror list, checks that the link is correct, and clicks the link element.

```{r}
open_url("https://www.r-project.org/")

s(".row") |>
  find_element("div") |>
  find_elements("a") |>
  elem_find(has_text("CRAN")) |>
  elem_expect(attr_contains("href", "cran.r-project.org")) |>
  elem_click(wait_for_navigation = TRUE)
```

Now that we're in the mirror list page, let's find the link to every CRAN mirror in the UK.

```{r}
s("dl") |>
  find_elements("dt") |>
  elem_find(has_text("UK")) |>
  find_element(xpath = "./following-sibling::dd") |>
  find_elements("tr") |>
  elem_expect(has_at_least(1)) |>
  as.list() |>
  lapply(
    \(x) x |>
      find_element("a") |>
      elem_attr("href")
  )
```

## Vignettes

* Get started with selenider, and learn the basics: `vignette("selenider")`.
* Use selenider with testthat, shinytest2 and Github Actions: `vignette("unit-testing", package = "selenider")`.
* Use selenider with rvest: `vignette("with-rvest", package = "selenider")`

Owner

Name: Ashby Thorpe
Login: ashbythorpe
Kind: user

Repositories: 3
Profile: https://github.com/ashbythorpe

R enthusiast

CodeMeta (codemeta.json)

{
  "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
  "@type": "SoftwareSourceCode",
  "identifier": "selenider",
  "description": "A user-friendly wrapper for web automation, using either 'chromote' or 'selenium'. Provides a simple and consistent API to make web scraping and testing scripts easy to write and understand. Elements are lazy, and automatically wait for the website to be valid, resulting in reliable and reproducible code, with no visible impact on the experience of the programmer.",
  "name": "selenider: Concise, Lazy and Reliable Wrapper for 'chromote' and 'selenium'",
  "relatedLink": [
    "https://ashbythorpe.github.io/selenider/",
    "https://CRAN.R-project.org/package=selenider"
  ],
  "codeRepository": "https://github.com/ashbythorpe/selenider",
  "issueTracker": "https://github.com/ashbythorpe/selenider/issues",
  "license": "https://spdx.org/licenses/MIT",
  "version": "0.4.1",
  "programmingLanguage": {
    "@type": "ComputerLanguage",
    "name": "R",
    "url": "https://r-project.org"
  },
  "runtimePlatform": "R version 4.4.2 (2024-10-31)",
  "provider": {
    "@id": "https://cran.r-project.org",
    "@type": "Organization",
    "name": "Comprehensive R Archive Network (CRAN)",
    "url": "https://cran.r-project.org"
  },
  "author": [
    {
      "@type": "Person",
      "givenName": "Ashby",
      "familyName": "Thorpe",
      "email": "ashbythorpe@gmail.com",
      "@id": "https://orcid.org/0000-0003-3106-099X"
    }
  ],
  "copyrightHolder": [
    {
      "@type": "Person",
      "givenName": "Ashby",
      "familyName": "Thorpe",
      "email": "ashbythorpe@gmail.com",
      "@id": "https://orcid.org/0000-0003-3106-099X"
    }
  ],
  "maintainer": [
    {
      "@type": "Person",
      "givenName": "Ashby",
      "familyName": "Thorpe",
      "email": "ashbythorpe@gmail.com",
      "@id": "https://orcid.org/0000-0003-3106-099X"
    }
  ],
  "softwareSuggestions": [
    {
      "@type": "SoftwareApplication",
      "identifier": "chromote",
      "name": "chromote",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=chromote"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "jsonlite",
      "name": "jsonlite",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=jsonlite"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "knitr",
      "name": "knitr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=knitr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "purrr",
      "name": "purrr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=purrr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "rmarkdown",
      "name": "rmarkdown",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rmarkdown"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "RSelenium",
      "name": "RSelenium",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=RSelenium"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "rvest",
      "name": "rvest",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rvest"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "selenium",
      "name": "selenium",
      "version": ">= 0.1.3",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=selenium"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "shiny",
      "name": "shiny",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=shiny"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "shinytest2",
      "name": "shinytest2",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=shinytest2"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "showimage",
      "name": "showimage",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=showimage"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "testthat",
      "name": "testthat",
      "version": ">= 3.0.0",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=testthat"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "wdman",
      "name": "wdman",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=wdman"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "xml2",
      "name": "xml2",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=xml2"
    }
  ],
  "softwareRequirements": {
    "1": {
      "@type": "SoftwareApplication",
      "identifier": "R",
      "name": "R",
      "version": ">= 2.10"
    },
    "2": {
      "@type": "SoftwareApplication",
      "identifier": "cli",
      "name": "cli",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=cli"
    },
    "3": {
      "@type": "SoftwareApplication",
      "identifier": "coro",
      "name": "coro",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=coro"
    },
    "4": {
      "@type": "SoftwareApplication",
      "identifier": "curl",
      "name": "curl",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=curl"
    },
    "5": {
      "@type": "SoftwareApplication",
      "identifier": "lifecycle",
      "name": "lifecycle",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=lifecycle"
    },
    "6": {
      "@type": "SoftwareApplication",
      "identifier": "prettyunits",
      "name": "prettyunits",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=prettyunits"
    },
    "7": {
      "@type": "SoftwareApplication",
      "identifier": "rlang",
      "name": "rlang",
      "version": ">= 1.1.0",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rlang"
    },
    "8": {
      "@type": "SoftwareApplication",
      "identifier": "utils",
      "name": "utils"
    },
    "9": {
      "@type": "SoftwareApplication",
      "identifier": "vctrs",
      "name": "vctrs",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=vctrs"
    },
    "10": {
      "@type": "SoftwareApplication",
      "identifier": "withr",
      "name": "withr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=withr"
    },
    "SystemRequirements": null
  },
  "fileSize": "660.182KB",
  "releaseNotes": "https://github.com/ashbythorpe/selenider/blob/master/NEWS.md",
  "readme": "https://github.com/ashbythorpe/selenider/blob/main/README.md",
  "contIntegration": [
    "https://github.com/ashbythorpe/selenider/actions/workflows/R-CMD-check.yaml",
    "https://app.codecov.io/gh/ashbythorpe/selenider?branch=main"
  ],
  "keywords": [
    "r",
    "web-scraping"
  ]
}

GitHub Events

Total

Create event: 6
Release event: 1
Issues event: 27
Watch event: 13
Delete event: 9
Issue comment event: 30
Push event: 53
Pull request event: 16
Pull request review comment event: 1
Pull request review event: 3

Last Year

Create event: 6
Release event: 1
Issues event: 27
Watch event: 13
Delete event: 9
Issue comment event: 30
Push event: 53
Pull request event: 16
Pull request review comment event: 1
Pull request review event: 3

Issues and Pull Requests

Last synced: 10 months ago

All Time

Total issues: 33
Total pull requests: 13
Average time to close issues: 25 days
Average time to close pull requests: 21 days
Total issue authors: 24
Total pull request authors: 2
Average comments per issue: 1.09
Average comments per pull request: 0.08
Merged pull requests: 8
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 19
Pull requests: 8
Average time to close issues: 22 days
Average time to close pull requests: 1 day
Issue authors: 16
Pull request authors: 1
Average comments per issue: 0.63
Average comments per pull request: 0.0
Merged pull requests: 6
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

ashbythorpe (4)
Aariq (3)
rcepka (3)
AntonioBergallo (3)
ercbk (1)
raphaludwig (1)
r2evans (1)
markwsac (1)
MitsuhaMiyamizu (1)
sybrohee (1)
mariusgordan (1)
hassaanuh (1)
artmg (1)
rgmerck (1)
nipnipj (1)

Pull Request Authors

ashbythorpe (12)
artmg (1)

Top Labels

Issue Labels

Pull Request Labels

Packages

Total packages: 1
Total downloads:
- cran 694 last-month

Total dependent packages: 0
Total dependent repositories: 0
Total versions: 5
Total maintainers: 1

cran.r-project.org: selenider

Concise, Lazy and Reliable Wrapper for 'chromote' and 'selenium'

Homepage: https://github.com/ashbythorpe/selenider
Documentation: http://cran.r-project.org/web/packages/selenider/selenider.pdf
License: MIT + file LICENSE
Latest release: 0.4.1
published over 1 year ago

Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 694 Last month

Rankings

Dependent packages count: 27.9%

Dependent repos count: 36.8%

Average: 50.3%

Downloads: 86.1%

Maintainers (1)

ashbythorpe@gmail.com

Last synced: 11 months ago

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science

selenider

Science Score: 26.0%

Keywords

Repository

Basic Info

Statistics

Topics

Metadata Files

README.Rmd

Owner

CodeMeta (codemeta.json)

GitHub Events

Total

Last Year

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

cran.r-project.org: selenider

Rankings

Maintainers (1)