webmockr) async requests via
either Async and/or AsyncVaried (thanks @keller-mark)
(#180)_PACKAGE pattern for pkg level man file
(#177)Via the two issues (#135) and (#159) crul now supports
http retries in: Async, AsyncVaried,
AsyncQueue, and the request builder class
HttpRequest
Async when urls
and disk vector lengths do not match (they must match)
(#165) thanks @shakrazAsyncQueue docs (#146) thanks @johnbaums !HttpClient docs to state that it’s an R6 class,
and give some details on what an R6 class is and links to more info
(#155)AsyncQueue gains methods: parse,
status_code, status, content, and
times (#156)$responses() method now returns an S3 class with an
associated print method to prevent printing a lot of results to the
screen; print method pritns a summary of results, and at most 10
results, just status code and url (#157)Paginator gains support for query parameter combination
page/per_page to automatically paginate
(#145)Paginator as well as for
HttpClient (#151) (#152) (#153) thanks @ateuchercrul would raise an error
when this header parsing happens, but now we raise a warning instead
(#150)ok() can now accept more than 1 status code so that you
can check if the status of a url is within a set of status codes rather
than equal to 1 status code (#124)ok() gains a parameter verb to use either
head or get requests. in addition added more documentation (#125) to the
function on how to get the “right answer” for whether a url is ok/up
(#123) (#127)ok() gains parameter ua_random, which if
TRUE, will use a random user agent string pulled from a
vector of 50 user agent strings generated from
charlatan::UserAgentProvider (#138)AsyncQueue for doing async
requests with rate limits (#139)curl_verbose() and
set_verbose(). curl_verbose() can be set by
passing to the initialize step (e.g.,
HttpClient$new(url, verbose=curl_verbose())), and gets more
compact verbose curl output, while also getting request body information
(and response body optionally). set_verbose() is sets
curl_verbose() globally (#141)HttpClient vs. Async)
(#133) (#143)AsyncVaried
class (#140)HttpRequest now adds the query (if present) to the
printed url in the print method for the class (it was absent before now)
(#128)delete-requests and post-requests
manual files - mostly redundant with other documentationHttpResponse response object gains new methods for
checking response content types, includes: raise_for_ct,
raise_for_ct_html, raise_for_ct_json,
raise_for_ct_xml. these behave similarly to
raise_for_status, and can behave as a warning or raise an
error through stop (#119) (#120)as.character.form_file when both
httr and crul are loaded (#112)head() verb on HttpClient was no
capturing auth when set on initialization (#122)jsonlite package moved to Imports (#112)parse() method in the HttpResponse
object now checks whether the response raw bytes can be converted to
character, and if not just returns raw bytes (#115) (#116)cainfo curl option, fixes problem
arising from change in recent libcurl version (#117)parse()
method of an HttpResponse class to the internally called
function iconv() to finely control the usage of
iconv for cases in which normal encoding conversion doesn’t
work (#110)httpcode package instead of fauxpas
package within ok() function (#108) (#109) thanks @maelle !HttpClient. both request and
response hooks supported. not supported in async methods for now (#76)
(#107)$parse() behavior (in the
HttpResponse object) when using disk or stream.
$parse() was throwing a warning when using disk and an
error when using stream. and improves behavior when doing async requests
(#104)Paginator gains optional progress bar through the new
progress parameter. In addition, the cat()
calls inside the method were removed, so as not to insert newlines with
each page and to not print “OK” when done (#106) thanks @boshekAsync (#101)
(#103)AsyncVaried with
curl of a certain version, works now (#102) (#103)HttpClient gains a retry method: retries
any request verb until successful (HTTP response status < 400) or a
condition for giving up is met. (#89) (#95) thanks @hlappHttpClient, HttpRequest, and
Async classes gain verb method for doing HTTP
requests specifying any of the supported HTTP verbs (#97)HttpClient and Paginator gain a
url_fetch method: get the URL that would be sent in an HTTP
request without sending the HTTP request. Useful for getting the URL
before executing an HTTP request if you need to check something about
the URL first. (#92)HttpResponse class as
$response_headers_all as an unnamed list, with each element
a named list of headers; the last list in the set is the final response
headers that match those given in the $response_headers
slot (#60) (#99)url_parse: lacked check that input was a string,
and that it was length 1 - this PR fixed that (#100) thanks @aaronwolenHttpStubbedResponse was removed from the package - it
may have been used at some point, but is not used in the package anymore
(#88)Async and AsyncVaried now support simple
auth, see ?auth (#70)ok() to ping a URL to see if it’s up
or not, returns a single boolean (#71) (#73)HttpClient and HttpRequest gain new
parameter progress that accepts a function to use to
construct a progress bar. For now accepts httr::progress()
but will accept other options in the future (#20) (#81)set_auth(), set_headers(),
set_opts(), set_proxy(), and
crul_settings() (#48) (#85)httpcode::http_code (#80)Async and
AsyncVaried to print max of 10 and tell user how many total
and remaining not shown (#72)proxy() for socks, e.g. to use with
TOR (#79)Async and AsyncVaried requests
fail, they don’t error but instead we capture the error and pass it back
in the result. this way any failure requests don’t stop progress of the
entire async request suite (#74) (#84)Paginator - it wasn’t handling pagination
correctly. In addition, fixed to hopefully handle all scenarios now.
added more tests (#62)urltools::url_encode to encode strings, but it wasn’t
encoding correctly in some locales. Using curl::curl_escape
fixes the problem. Encoding is done on query values and names (#67)
(#68)Paginator to help users
automatically paginate through multiple requests. It only supports query
parameter based paginating for now. We’ll add support later for other
types including cursors (e.g., used in Solr servers), and for link
headers (e.g., used in the GitHub API). Please get in touch if you find
any problems with Paginator. (#56)Async and Asyncvaried gain
ability to write to disk and stream data (to disk or elsewhere, e.g. R
console or to an R object) (#46) thanks @artemklevtsov for the push to do
thisauth to indicate that
user and pwd are indeed required - and to
further indicate that one can pass in NULL to those
parameters (similar to an empty string "" in
httr::authenticate) when one e.g. may want to use
gssnegotiate method (#43)I() (#55)head requests with HttpClient
when passing query parameter - it was failing previously.
Added query parameter back. (#52)upload() and
examples (#25)crul now drops any options passed in to
opts or to ... that are not in set of allowed
curl options, see curl::curl_options() (#49)?cookies for how to set cookies
(#44)disk and stream from
head method in HttpClient and
HttpRequest as no body returned in a HEAD requestAsyncVaried to return async responses in the
order that they were passed in. This also fixes this exact same behavior
in Async because Async uses
AsyncVaried internally. (#41) thanks @dirkschumacher for reportingwebmockr, which is now on CRAN.auth() to do simple authentication
(#33)HttpStubbedResponse for making a stubbed
response object for the webmockr integration (#4)mock() to turn on mocking - it’s off by
default. If webmockr is not installed but user attempts to
use mocking we error with message to install webmockr
(#4)gzip-deflate by deafult for each request to make
sure gzip compression is used if the server can do it (#34)useragent to User-Agent as default
user agent header (#35)url_build and
url_parse (#31)AsyncVaried (#30)crul in realistic scenarios
rather than brief examples to demonstrate individual features (#29)HttpRequest (#28)HttpClient (#26)Async and AsyncVaried. The former
being a simpler interface treating all URLs with same options/HTTP
method, and the latter allowing any type of request through the new R6
class HttpRequest (#8) (#24)HttpRequest to support
AsyncVaried - this method only defines a request, but does
not execute it. (#8).... And we check that user
doesn’t pass in prohibited options (curl package takes care
of checking that options are valid) (#5)fauxpas package for dealing with HTTP
conditions. It’s a Suggest, so only used if installed (#6)curl::curl_fetch_stream. stream param defaults
to NULL (thus ignored), or pass in a function to use
streaming. Only one of memory, streaming or disk allowed. (#9)curl::curl_fetch_disk.
disk param defaults to NULL (thus ignored), or
pass in a path to write to disk instead of use memory. Only one of
memory, streaming or disk allowed. (#12)raise_for_status() method on the
HttpResponse class (#10)httpcode but wasn’t using it in the
package. Now using the package in HttpResponse