You are here

Using ERDDAP with NCEI Model Data

ERDDAP is a visualization and data conversion tool that can work from various data sources. In the case of NCEI model data, the data sources are THREDDS aggregations of those models.

The ERDDAP front page, shown in figure 1, gives an overview of information about ERDDAP and what data is available, along with search options with descriptions. It also has links to assorted tools for working with data. All ERDDAP deployments come bundled with built-in documentation. Detailed information about the version of ERDDAP that NCEI is using is available on the NCEI ERDDAP Server’s ERDDAP>Information page.

The Pacific Fisheries Environmental Laboratory developed ERDDAP.

Finding Data

ERDDAP’s front page features a link to a full list of datasets, a search box, and links to datasets that follow several protocols. Near the top left of the main page, there is a link that reads "View a list of All ## Datasets", where "##" is the number of datasets. Clicking this link will show a table of all available datasets. A search box labeled “Do a Full Text Search for Datasets” is below that link. Enter search terms in this box and click "Search" to show a table of all datasets having titles, variables, or metadata matching those terms. Figure 2 shows the table that results from searching for "CFS."

Dataset Types

ERDDAP Datasets can be either gridded data or table data, referred to in the ERDDAP UI as "GridDAP," which is an acronym for "Grid Data Access Protocol", and "TableDAP", which is an acronym for "Table Data Access Protocol."

GridDAP datasets are multidimensional grids with shared dimensions. They typically have time, vertical, north-south, and east-west dimensions, or any combination of these. The horizontal dimensions may either be geographic or arbitrary Cartesian coordinates. All NCEI data currently in the NCEI-NC ERDDAP is grid data. The NCEI-MS server uses some TableDAP data.

TableDAP data sets look like a spreadsheet or database table. You can query them in a similar way to database tables. While the NCEI-NC ERDDAP server does not host any table data, ERDDAP will always make the dataset list itself available as a TableDAP dataset.

The Datasets List Table

Searching, selecting all datasets, or selecting datasets by type all show a Datasets List HTML table, shown in Figure 2, as a result. This table has a row for each data set containing information about the sets, links to more information, and links for downloading and graphing the data. The columns are as follows:

  • GridDAP Data: This column contains links to a page for selecting and downloading gridded data. These links appear only for GridDAP data sets.
  • Sub-Set: This column contains links to a page that allows you to subset data to select in the TableDAP data access page. That page is not enabled for any NCEI datasets.
  • TableDAP Data: This column contains links to a page for selecting and downloading table data. These links only appear for TableDAP data sets.
  • Make A Graph: This column contains links to a page for viewing graphs of data and downloading the data used for the graphs.
  • WMS: This column contains links to a web map page that provides instructions for using ERDDAP as a Web Mapping Service (WMS) for that dataset. Most NCEI datasets do not enable this feature.
  • Title: This column contains descriptive titles for each dataset.
  • Summary: This column contains question mark icons that show information about each dataset when the pointer is over that icon.
  • Meta-data: This column contains links to metadata. There is always a link "M" for properties of the dataset that come from its original source or are added as part of the ERDDAP configuration. This metadata is presented as an HTML table that can optionally be downloaded as JSON. If the ERDDAP configuration contains the proper files or has metadata conversion enabled, links for FGDC and ISO 19115 metadata appear as linked “F” or “I”.
  • Background Info: This column contains links with information about the dataset. In the case of NCEI model data, it is usually the THREDDS catalog page or the OpenDAP data selection page.
  • RSS: This column contains links to RSS feeds for each dataset that contain notifications about updates to the data.
  • Institution: This column lists the institutions that provides the data.
  • Dataset ID: This column contains a unique identifier for each dataset.

JSON Representation of Dataset Lists

All HTML tables available in the ERDDAP user interface are also available in JSON format by changing the ".html" in the URL that led to them to ".json". For example, the list of CFS datasets shown in Figure 2 were accessed with the URL https://www.ncei.noaa.gov/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=CFS. The URL for that same list in JSON is: https://www.ncei.noaa.gov/erddap/search/index.json?page=1&itemsPerPage=1000&searchFor=CFS.

The JSON format is a dictionary with an entry “table” that is a dictionary that contains the headings and data for the requested dataset list. This dictionary has the following entries:

  • columnNames: This is a list of the names of the columns in the table
  • columnTypes: This is a list of the data type of each column. For the dataset list these are all strings, but they can be other types for metadata, data, and other types of tables.
  • rows: This is a list of data in lists that order each item in the same position as the name of it in the "columnNames" list.

Links in the datasets table will be strings of their URL. Links that would have gone to ERDDAP HTML pages are modified to go to a JSON version of that page. The "GridDAP" and "TableDAP" links will not have the ".html" extension on them that opens the data access page; they will be just the base URL of the data that can have data type extensions added to it to do different data conversions. The summary column is the full text of the summary rather than a link to it, as it is in the HTML version. The "graph" links will still link to the graphing UI for the dataset. The "wms" links to the root of the request URL for the WMS service rather than the web map example.

Metadata links will be in separate columns in the JSON list, with the headings “Info” for the attributes table, “FGDC” for the FGDC metadata and “ISO 19115” for ISO metadata.

Viewing and Downloading Data

The Graph Page

The graph page, shown in figure 3, allows you to configure and produce graphs. You can also download data based on what is shown in the graph using this interface. The interface is a set of controls for defining the graphs with a preview graph. Because NCEI model data always uses GridDAP protocol, this article will discuss the interface for gridded data.

Help icons appearing as circled question marks appear throughout the interface; these have tooltips that give more information about the item that they are next to. Help icons near user interface widgets describe the widget, help icons near dimension names show the size and attributes of the dimension, and help icons near the "summary" and "license" part of the "information" section will show the summary and license. Some widgets have tooltips that appear when you mouse over the widgets.

The Preview Graph

The preview graph is on the upper right side of the graph page. When the page first appears, it will try to guess the most proper graph for the data provided. For example, data with latitude and longitude displays as a map, while data that represents a time series displays as a line graph. The UI has a series of controls that let you select which variables to use as dimensions and data, and what range of dimensions to use. While changing some options will cause the graph to immediately update, you usually click the "Redraw the Graph" button to update the graph.

You can click on the preview graph to recenter and zoom in on the image. Click the zoom buttons above the preview graph to select different levels of zoom. Each button has a tooltip describing its function; A question mark icon near these buttons will show a brief description of the map controls when when the mouse pointer is over them.

Dimensions

The "dimensions" section of the interface allows you to select a value or range of values for each dimension in the variable. Each dimension has a slider widget. These can represent a range or a single value depending on the type of graph selected. Dimension sliders for a single value will have one pointer than can be moved and a box for directly entering the dimension value. Dimension sliders that represent a range will have two half-circle shaped pointers along with a text box for the start and end of the range. The dimension text boxes have buttons next to them to increment or decrement the dimensions, or set them to their first or last value. Tooltips for the dimension text inputs show the valid range, number of values, and other information for that dimension. The buttons next to those boxes have tooltips describing what they do.

The dimensions section will have only dimensions relevant to the current graph type and selected dimensions for the graph. The graph type and selected dimensions determine which dimension sliders allow you to select a range or a single value.

Graph Type

The "Graph Type" select box controls how the graph will be displayed and which options appear in the rest of the UI. You can select which data to use for axes, colors, and vectors using select boxes directly below the Graph type select box. The following options for Graph Type are available:

  • Surface: This will show an image map for 2-d data. You can select variables for the x-axis, y-axis, and color. The axes must be dimension variables while the color must be a multidimensional variable. If the x and y axis are latitude and longitude, the data will be displayed with world map of countries. The default values for x and y are the horizontal and vertical components if ERDDAP can determine them, while the color is typically the first multidimensional variable alphabetically. You can set the dimensions selected for x and y axes to a range of values, while you can only set other dimensions to a single value.
  • Vectors: This graph shows arrows representing data with direction and magnitude such as wind or water current. Two multi-dimensional data variables define the size and direction of the arrows. One of those represents the x components of the vectors while the other represents the y components. You can select variables for the x-axis, y-axis, vector X, and vector Y. The axes must be dimensions. ERDDAP will try to default the vector X and vector Y variables to be the first component-style (u,v) variables it can find.
  • Lines: This is a simple line graph. You can select variables for x-axis and y-axis; the x-axis is a dimension, while the y-axis is a data variable, which may be multidimensional. You can select a range of values for the dimension used as the x-axis, while other dimensions can be set to a single value.
  • Markers: This is similar to the lines option, but markers at the data values appear rather than lines connecting them. In addition to selecting the axes, you can select a variable to define the color of the markers.
  • Lines and Markers: This is a combination of the Lines type and the Markers type.
  • Sticks: This is a sequence of lines which point in a direction controlled by two of the data variables. You can use this graph type for things such as current direction over time. This type lets you select the x axis and the “Stick X” and “Stick Y” variables, which will define the direction the stick points in the graph. The dimension for the x axis can have a range of values selected for it, while the other dimensions can have a single value selected.

Graph Settings

Graph settings allow you to define the color, style and data ranges of various components of the graph.

The following graph controls are available for graphs where they are relevant:

  • Color: This is the color of markers, lines, or vectors in a graph.
  • Color Bar: This selects the color scale to use for pixels in a surface plot or markers in marker or line and marker graphs where color is assigned to a data variable. This control has a list of available color schemes. The color bar control has the following controls describing how the colors are interpreted:
    • Continuity: This describes whether the color bar is smooth or banded with these options:
      • Continuous: When you select this option, the graph generator will color the pixels or markers by interpolating between the nearest values of the color bar.
      • Discrete: When you select this option, the graph generator will color the pixels or markers to match a specific value range in the color bar.
    • Scale: This determines whether the colors have a linear or logarithmic relation to the data.
    • Min: This sets the data value that the first color in the color bar will represent.
    • Max: This sets the data value that the last color in the color bar will represent.
    • N Sections: This sets the number of sections in the color bar.
  • Draw the land mask: This controls whether the land mask covers data or not with these options:
    • Under the data: This causes the data to cover the land mask if it is over land.
    • Over the data: This causes only ocean and sea data to be visible.
  • Marker Type: This combo box allows you to select the shape of markers on graphs that use them.
    • Size: This sets the size of the markers.
  • Y Axis: This lets you control the range and orientation of the Y axis
    • Maximum: This sets the maximum Y value to be shown on a graph.
    • Minimum: This sets the minimum Y value to be shown on a graph.
    • Ascending or Descending: This combo box lets you pick whether the maximum value is at the top or bottom of the graph.
  • Vector Standard: This specifies how many data units will result in a vector that matches the sample vector in a vector graph’s legend.

Download Controls

The download controls, which are below the graph settings, let you select the final format to download and have a "Download the Data or an Image" button. The format selector uses file extensions to describe formats. Selecting file extensions describing image or document formats will cause an image to be downloaded, while selecting other options will cause the data itself to be downloaded. While there is a wide selection of formats, not all formats support all variations of dimension values and variable requests. For example, when using the ".esriASCII" option you can only download 1 variable which must be 2 dimensional, while you can download NetCDF with any selections that are available for the graph.

The download controls include a URL text box that will contain a URL that will download the current graph data in the selected format. You can copy this URL for use in client software. Links next to these boxes include instructions on how to use these URLs along with information about each file type.

The Grid Data Access Form

The data access page allows you to select variables from the dataset and ranges for all dimensions. This section will discuss data access for GridDAP, which all NCEI model data uses.

Dimensions

The "dimensions" section of the interface allows you to select a value or range of values for each dimension in the variable. Each dimension has a slider widget representing a range. Dimension sliders have two half-circle shaped pointers along with a text box for the start and end of the range. GridDAP datasets will allow ranges for all dimensions. The dimension text inputs show the valid range, number of values, and other information for that dimension.

Variables

The grid variables section has a list of variables in the file alongside checkboxes so that you can select variables that will be in the file to download. Each one has a question mark icon next to it that shows a declaration for that variable in a style similar to C++ or JAVA.

Download Controls

The download controls, which are below the variable list, let you select the final format to download and have a "Submit" button for starting the download. The format selector uses file extensions to describe formats. Selecting file extensions describing image or document formats will cause an image to be downloaded, while selecting other options will cause the data itself to be downloaded. While there is a wide selection of formats, not all formats support all variations of dimension values and variable requests. For example, the ".esriASCII" format can only represent 1 variable which must be 2 dimensional, while NetCDF can contain any selections that are available for the data. ERDDAP will report an error if you select a graph format using a data selection that none of the graph types supported by ERDDAP can represent. Clicking the "more info" link next to the format selector will show information on the available formats.

The download controls include a URL text box with a "Just generate the URL" button next to it. Clicking that button will populate the URL text box with a URL that can be used to download the data according to the current settings. You can copy that URL for use in client software. A link next to this box includes instructions on how to use these URLs.

The HTTP Get Interface

Any process that you can run with the graph or download page is possible with a Representational State Transfer (RESTful) interface using HTTP get requests.

For more information about the HTTP interface for gridded data, consult the ERDDAP GridDAP Documentation.

Example Processes

You can learn how to use ERDDAP for specific tasks on our ERDDAP Example Process Instructions page.