There are a number of different crul classes that do HTTP requests. The following table compares features across the classes.
| Class | HTTP verbs | Asynchronous? | Packges using |
|---|---|---|---|
| HttpClient | all | no | bcdata,chirps,duckduckr,gfonts,mindicador,nsapi,tradestatistics,viafr |
| Paginator | all except retry | no | — |
| ok | head,get | no | dams |
| Async | all except retry | yes | rdryad,rnoaa |
| AsyncVaried | all except retry | yes | rcrossref,taxize,mstrio |
| AsyncQueue | all except retry | yes | — |
HttpClient is the main class for doing synchronous
HTTP requests. It supports all HTTP verbs including retry. It was the
first class in this package. See also crul::ok(), which
builds on this class.
Paginator is a class for automating pagination. It
requires an instance of HttpClient as it’s first parameter.
It does not handle asynchronous requests at this time, but may in the
future. Paginator may be the right class to use when you
don’t know the total number of results. Beware however, that if there
are A LOT of results (and a lot depends on your internet speed and the
server response time) the requests may take a long time to finish - just
plan wisely to fit your needs.
ok is a convienence function light wrapper around
HttpClient. It’s use case is for determining if a URL is
“up”, or “okay”. You don’t have to instantiate an R6 class as you do
with the other classes discussed here, but you can pass an
HttpClient class to it if you like.
With Async you can make HTTP requests in parallel.
Async does not at this time support retry. It’s targeted at
the use case where you don’t mind having request settings the same for
all requests - you just pass in any number of URLs and all requests get
the same headers, auth, curl options applied, if any.
AsyncVaried allows you to customize each request
using HttpRequest (See below); that is, every HTTP request
run asynchronously can have its own curl options, headers, etc.
AsyncQueue is the newest class, inheriting from
AsyncVaried, introduced in curl v1. The motivation behind
this class is: sometimes you may want to do HTTP requests in parallel,
but there’s rate limiting or some other reason to want to not simply
send off all requests immediately. AsyncQueue sets up a
queue, splitting up requests into buckets, and executes requests based
on a sleep or req_per_min (requests per
minute) setting.
HttpRequest is related here, but not in the table above
because it doesn’t do actual HTTP requests, but is used to build HTTP
requests to pass in to AsyncVaried. The simplified class
Async relative to AsyncVaried uses
HttpRequest internally to build requests.
See the async with crul vignette for more details on asynchronous requests.