--- title: "Getting Started" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Getting Started} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ## Intro This package was designed to make it easier to work with both [ERA5-Land](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-land) and [AmeriFlux](https://ameriflux.lbl.gov) data, hence the name "ERA5Flux". There may be instances where the AmeriFlux data contain gaps that you want to fill with ERA5-Land data to create a single time series for your analysis. Please note that this package only merges and gap fills for solar radiation (long name: "surface_solar_radiation_downwards" / short name: "ssrd"), air temperature (long name: "2m_temperature" / short name: "t2m"), and precipitation (long name: "total_precipitation" / short name: "tp"). These variables are defined in the [ERA5-Land documentation here](https://confluence.ecmwf.int/spaces/CKB/pages/140385202/ERA5-Land+data+documentation). Here is a demonstration of the workflow needed to achieve that result. ## Workflow ### Step 1: Get AmeriFlux data, AmeriFlux site metadata, and ERA5-Land Data First we will need to download AmeriFlux data. Navigate to [AmeriFlux](https://ameriflux.lbl.gov) and login to your account. Once you're logged in, you can download data at the [Download page](https://ameriflux.lbl.gov/data/download-data/). You will be presented with options on the types of data products you would like to download. Select **AmeriFlux BASE** then select **CC-BY-4.0** as the data use policy that you will follow. At the next step, pick the sites that you would like, then write a short description of your intended use for this data and agree to the license. Finally, you will be presented with a data download page that has download links for the requested sites. Click on those download links to get a zip file for each site. You will also need to click on the "Requested_Files" download link to get the metadata text file for your request. This will download as "requested_files_manifest_YYYYMMDD.txt". After you've downloaded everything, unzip the site zip files. To ensure that the rest of the workflow runs smoothly, we recommend that you arrange your files like so: ``` your-working-directory/ ├── unzipped_AmeriFlux_data/ │ ├── unzipped_site1_folder/ │ │ ├── *.csv │ │ └── *.xlsx │ │ │ ├── unzipped_site2_folder/ │ │ ├── *.csv │ │ └── *.xlsx │ │ │ ├── unzipped_site3_folder/ │ │ ├── *.csv │ │ └── *.xlsx │ │ │ └── requested_files_manifest_YYYYMMDD.txt │ ├── site1_data.zip ├── site2_data.zip ├── site3_data.zip │ └── more stuff/ ├── ... └── ... ``` After arranging the files, you can now generate the AmeriFlux site metadata. Specify the name(s) of the ERA5-Land variable(s) you would like to get. Since we already got all the AmeriFlux variables, we need the corresponding names to the same variables in ERA5-Land that we're interested in. In this example, we specified `surface_solar_radiation_downwards` as the ERA5-Land variable we're interested in. For the purposes of this vignette, the file path `system.file("extdata", "example_AmeriFlux", package = "ERA5Flux")` points to a folder that contains the example unzipped site folders and example requested files manifest. Please replace this file path with your own path to the AmeriFlux data you previously unzipped. Once you point to your own folder of unzipped AmeriFlux data, run `get_site_metadata()`. The code below is a demonstration of what this part may look like. ```{r setup} # Load the package library(ERA5Flux) # Specify the ERA5-Land variables you want to get my_variables <- c("surface_solar_radiation_downwards") # Point to the folder containing the unzipped site folders and requested files manifest # For the purposes of this demo, an example data folder will be used # Please point to your own existing folder for your own workflow my_AmeriFlux_folder <- system.file("extdata", "example_AmeriFlux", package = "ERA5Flux") # Generate the AmeriFlux site metadata file my_site_metadata <- get_site_metadata(folder = my_AmeriFlux_folder, selected_variables = my_variables) my_site_metadata ``` The metadata data frame will have 5 columns: `site_codes`, `lat`, `lon`, `startdate`, `enddate`, and `variables`. We will need this metadata in order to build our ERA5-Land data requests. After you generate the AmeriFlux site metadata data frame, you can now focus on getting the corresponding ERA5-Land data. `download_ERA5()` requires a land-sea mask so you must download that first with `get_land_sea_mask()`. This function will download the land-sea mask from [here](https://confluence.ecmwf.int/pages/viewpage.action?pageId=140385202#ERA5Land:datadocumentation-parameterlistingParameterlistings). ```{r, eval = F} # Download the land-sea mask if you haven't done so already get_land_sea_mask() ``` Then make sure you create an account at the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/) if you haven't done so already. You may need to accept the data license agreement first before you can download the data. Visit the [Climate Data Store User Profile page](https://cds.climate.copernicus.eu/profile) to accept the appropriate license(s). `download_ERA5()` also requires a valid Climate Data Store API key. When you're logged into the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/), you can grab your API key by clicking on your name for your account in the top right corner, selecting "Your profile", scrolling down to the "API key" section, and copying the API key. Copy and paste your Climate Data Store API key into the `my_key` argument. Provide the AmeriFlux site metadata in the `site_metadata` argument, and set the file path to your land-sea mask as well. Finally, specify a folder to where you want to download your ERA5-Land data to. The folder should be an existing folder on your own machine. Please note that the download may take a while. ```{r, eval = F} # Paste your own API key my_key <- "my_own_ECMWF_key" # Set the path to your land-sea mask my_path_to_mask <- "lsm_1279l4_0.1x0.1.grb_v4_unpack.nc" # Point to the folder where you want the ERA5-Land data to download to # For the purposes of this demo, some example placeholder text is shown # Please point to your own existing folder for your own workflow my_ERA5_download_path <- "my_own_path_to_ERA5_download_folder" # Download the ERA5-Land data download_ERA5(my_key = my_key, site_metadata = my_site_metadata, mask = my_path_to_mask, download_path = my_ERA5_download_path) ``` If you have multiple sites, `download_ERA5()` will download files from every site into the folder you specify in `download_path`. Once the download finishes, it's recommended to organize your files so that you have a separate folder for each site. Make sure that each site folder has all the downloaded data files for its respective site. ### Step 2: Data Processing For this vignette, an example folder containing downloaded ERA5-Land data will be used (`system.file("extdata", "example_path_to_ERA5_download_folder", package = "ERA5Flux")`). Please remember to point to your own folder of downloaded ERA5-Land data. After downloading the ERA5-Land .nc files, we convert them into CSV files formatted to match AmeriFlux standards, enabling easy merging with existing datasets. This conversion is handled by `netcdf_df_formatter()`, which adjusts ERA5-Land timestamps from UTC to local time (yyyyMMddHHmm) based on each site’s coordinates and converts solar radiation (ssrd), air temperature (t2m), and total precipitation (tp) to AmeriFlux units. The `netcdf_to_csv()` function then calls `netcdf_df_formatter()` for each site folder, merging all relevant data by site and year so that each output CSV contains one site’s data for one year. If `full_year = TRUE`, the function will only return full years that begin with the first hour of the year and end with the last. For the purposes of this example, set `full_year = FALSE`. ```{r} # Point to a folder containing ERA5-Land .nc files # For the purposes of this demo, an example data folder will be used # Please point to your own existing folder for your own workflow site_folder <- system.file("extdata", "example_path_to_ERA5_download_folder", package = "ERA5Flux") # Create a temporary directory to export our output to output_filepath <- tempdir() # Specify a site name site_name <- "US_GL2" # Specify the site latitude and longitude coordinates site_lat <- as.numeric(my_site_metadata$lat) site_lon <- as.numeric(my_site_metadata$lon) # Convert netCDF data to a CSV file netcdf_to_csv(site_folder, output_filepath, site_name, site_lat, site_lon, full_year = FALSE) # Read the CSV back in data <- read.csv(list.files(output_filepath, pattern = "US_GL2", full.names = TRUE)) head(data) ``` ### Step 3: Merging and Blending AmeriFlux with ERA5-Land Data After you processed the ERA5-Land data, you can merge it with the AmeriFlux data. Remember to use your own file paths to your AmeriFlux and ERA5-Land data. Example data is used in this vignette. The `merge_ERA5_Flux()` function synchronizes AmeriFlux tower observations with ERA5-Land reanalysis data, ensuring both datasets share consistent timestamps and comparable variable names. It first reads AmeriFlux BASE data, replaces missing values with `NA`, and imports the processed ERA5-Land CSV file generated from `netcdf_to_csv()`. Because ERA5-Land data are typically hourly and AmeriFlux data are often half-hourly, the function linearly interpolates ERA5-Land variables to match the AmeriFlux time step. It then merges the datasets based on the aligned time column, adding corresponding AmeriFlux and ERA5-Land variables side by side. The resulting data frame provides a harmonized, site-specific time series that researchers can use to compare reanalysis and in situ flux measurements, fill short data gaps, and prepare blended climate–flux inputs for ecosystem or hydrological modeling. ```{r} # Point to AmeriFlux CSV data # For the purposes of this demo, an example data file will be used # Please point to your own existing file for your own workflow ameriflux_file <- system.file("extdata", "example_AmeriFlux", "AMF_US-GL2_BASE-BADM_2-5", "AMF_US-GL2_BASE_HH_2-5.csv", package = "ERA5Flux") # Point to ERA5-Land CSV data # For the purposes of this demo, an example data file will be used # Please point to your own existing file for your own workflow era5_file <- list.files(output_filepath, pattern = "US_GL2", full.names = TRUE) # List AmeriFlux variable(s) to be merged ameriflux_var <- c("SW_IN") # List ERA5-Land variable(s) to be merged era5_var <- c("ssrd") # Merge them together merged_data <- merge_ERA5_Flux(filename_FLUX = ameriflux_file, filename_ERA5 = era5_file, varname_FLUX = ameriflux_var, varname_ERA5 = era5_var) merged_data[3162:3182,] ``` As shown above, the merged data contains ERA5-Land data in `ssrd`, as well as AmeriFlux data in `SW_IN`. Finally, you can blend the merged data. The `blend_ERA5_Flux()` function combines AmeriFlux and ERA5-Land datasets into a single, gap-filled time series by applying flexible blending rules that determine how missing or incomplete AmeriFlux data should be replaced or adjusted using ERA5-Land values. Building on the merged output from `merge_ERA5_Flux()`, this function allows users to select among four blending approaches: simple replacement, linear regression with or without an intercept, or an automatic rule that adapts based on data completeness. For example, when more than half of the AmeriFlux values are available, the function uses a regression-based correction to preserve site-specific patterns while filling gaps; otherwise, it substitutes ERA5-Land data directly. The blended output adds new columns (e.g. `SW_IN_f`) that contain the harmonized variables, producing a continuous, high-quality dataset suitable for modeling, data assimilation, or long-term flux-climate analysis. The final result will look something like this: ```{r} # Specify the blending rule(s) # If you have multiple variables, specify a rule for each variable blending_rule <- c("replace") # Blend them together blended_data <- blend_ERA5_Flux(merged_data = merged_data, varname_FLUX = ameriflux_var, varname_ERA5 = era5_var, blending_rule = blending_rule) blended_data[3162:3182,] ``` In this example, the simple replacement blending rule was used, so a new `SW_IN_f` column was created with values from the ERA5-Land column, `ssrd`.