USGS - science for a changing world

Hypoxia in the Gulf of Mexico

USGS Info on MRB Nutrients
Gulf of Mexico Hypoxic Zone
Hypoxia Task Force
Other Agency Info
USGS Publications

Report Site Map > Sources and Preparation of Data Used for Nutrient Flux Estimation

USGS Open-File Report 2007-1080 - Streamflow and Nutrient Fluxes of the Mississippi-Atchafalaya River Basin and Subbasins for the Period of Record Through 2005

Sources and Preparation of Data Used for Nutrient Flux Estimation

Flux estimation uses daily streamflow data and nutrient concentrations from analysis of periodic water-quality samples. This narrative summarizes the sources of streamflow and water-quality data used for estimating long-term nutrient fluxes at sites in the Mississippi-Atchafalaya River Basin. It also describes the steps taken to prepare the nutrient concentration datasets. These steps include: (1) removal of inappropriate samples, (2) removal of zero concentrations, (3) censoring water-quality data as per USGS policies and procedures, (4) combining multiple nutrient parameters into one using standardized substitution rules, (5) changing laboratory reporting levels to long-term method detection levels; (6) screening for and removal of outliers, (7) averaging replicate concentrations into a single daily concentration, and (8) combining station data.

Specific information regarding the number of samples or individual concentrations for which each step was applied is not included. The nutrient species discussed in the following text are referred to in terms of their molecular formula or an abbreviation as indicated in table 1.

Table 1. Molecular formula or abbreviations used for nutrient species.

Nutrient Species    Molecular Formula or Abbreviation 
  Ammonia NH3
  Nitrate NO3
  Nitrite NO2
  Total Nitrogen TN
  Total Kjeldahl Nitrogen TKN
  Phosphorus P
  Orthophosphate OP
  Phosphate PO4
  Total Phosphorus TP

Data Sources for Streamflow Data

Stream discharge data were obtained from the USGS National Water Information System (NWIS) web-based database (NWISWeb). Stream discharge data for the lower part of the Mississippi-Atachfalaya River Basin were obtained from the U.S. Army Corps of Engineers (USACE) New Orleans District Water Control Section website. Daily flow data for the Red River at Alexandria, Louisiana, for the period 1983 to 2005 were obtained directly from the USACE Vicksburg, Miss. office. Daily flow data for the Tennessee River at highway 90 near Paduchah, Kentucky, were obtained directly from the Tennessee Valley Authority.

Data Sources for Water-Quality Data

Water-quality samples were collected by the USGS using consistent nationwide protocols (U.S. Geological Survey, variously dated). Most of the water-quality analyses were performed by the USGS. Some analyses were performed at other laboratories which the USGS had cooperative agreements with, and data from these laboratories were approved by the USGS. Information regarding USGS methods used for analyzing water-quality parameters are available from the National Environmental Methods Index database (USGS inorganic methods search).

Data for all water-quality parameters of interest for the period of record were retrieved for USGS water-quality monitoring stations from the host USGS Water Science Center (WSC) National Water Information System (NWIS) quality of water database (QWDATA). Different WSCs maintain different stations and store the data from those stations independently. Data also were acquired from the USGS National Stream Quality Accounting Network (NASQAN), specifically since water year (WY) 1996 (10/1/95). In a few cases, samples or individual analyses were not present in both of the databases, therefore the data were merged creating the most complete dataset. There were four individual analytical results out of the thousands used in this analysis that differed between the two datasets that could not be explained by rounding, and in each of these cases, the concentration from the NASQAN dataset was used.

Data Screening and Preparation Steps

1. Removal of inappropriate samples

Samples were removed if they (1) did not contain any water-quality data for the parameters of interest, (2) were composite samples (these typically are samples collected over several days), (3) were individual cross-sectional samples (these are samples taken in sequence across a river), or (4) did not have the appropriate medium type or sample type code. Samples were included in the analysis only if they had a medium code of 9 (surface-water sample), 0 (medium code not determined), or R (replicate). Samples were included in the analysis only if they had a sample type code of 9 (regular sample), 5 (duplicate), 7 (replicate), or A (sample type code not determined).

2. Removal of zero concentrations

Data collected prior to 1982 in rare cases had zero concentrations for individual water-quality parameters. The user’s manual for QWDATA indicates that zero values can be recensored to the minimum reporting limit for several of the parameters used in this analysis (section 4.9, appendix I., Parameters that can be recensored if stored value is zero; U.S. Geological Survey, 2006). But a previous analysis using much of the same data to estimate nutrient fluxes in the Mississippi-Atchafalaya River Basin (Goolsby and others, 1999) indicated that zero concentrations were more likely an indication of a missing value and were removed.

An analysis of the data used in this study indicates that for parameters with zero values, when there was a concentration available for another similar water-quality parameter, about one-third of the time the other value was also zero, about one-third of the time the other value was a censored value, and about one-third of the time the other value was above the detection limit. Because of the significant number of cases of uncensored values for similar parameters, it is unreasonable to assume that zero values always represent censored values. Therefore, zero concentrations were assumed to be missing and were not used in this analysis.

3. Recensored values for NH3 and TP as per USGS Memorandums

USGS National Water-Quality Laboratory (NWQL) Technical Memorandum 97.10 (Patton, 1997) states that dissolved NH3 data with values of < 0.01 mg-N/L, 0.01 mg-N/L, and < 0.015 mg-N/L should be recensored as < 0.02 mg-N/L for samples analyzed before 10/1/1977. All NH3 concentrations with values below 0.02 mg-N/L for both parameter codes 00608 and 00610 collected before 10/1/1977 were changed to < 0.02 mg-N/L.

USGS NWQL Technical Memorandum 98.07 (Foreman and others, 1998) states that TP data from < 0.01 mg-P/L to MRL of 0.03 mg-P/L for analyses done between 10/1/91 and 9/30/98 that were analyzed using the methods used per Office of Water-Quality Technical Memorandum 92.10 (method codes C and D; U.S. Geological Survey, 1992) should be recensored to < 0.03 mg-P/L. All TP concentrations with parameter code 00665 and method codes of C or D with concentrations < 0.03 mg-P/L were changed to < 0.03 mg-P/L for period 10/1/91 - 9/30/98.

4. Combining multiple parameters into one individual constituent

Over the period of record, different water-quality parameters and analytical techniques have been employed by the USGS. There have been many analytical advancements over time, and the USGS has changed their methods of analyses to accommodate these improvements. Hence, different parameters must be combined to create a long-term constituent-specific dataset.

The USGS OWQ Technical Memorandum 93.04 (Rickert, 1992) supports the substitution of total NO3 + NO2 for dissolved NO3 + NO2, total NH3 for dissolved NH3, total orthophosphate for dissolved orthophosphate, and total NO2 for dissolved NO2. In summary, there was no difference in concentrations between filtered and “unfiltered” samples analyzed by the four-channel analyzer (which was used up through 1993) as “unfiltered” samples were analyzed from the supernatant of a well-settled sample (and may have even been filtered at the analyst’s discretion for turbid samples, but was still coded as “unfiltered”). Furthermore, a digestion step is really necessary in order to analyze these nutrients for total determinations.

In many cases in the NWIS QWDATA database, concentrations for one parameter code are calculated from analyzed values of other parameter codes (by either summing or taking the difference of measured concentrations) to either change units or to determine individual or combined chemical species. The QWDATA manual lists which parameters are calculated parameters in section 4.4, Appendix D (Calculated parameters; U.S. Geological Survey, 2006). In some cases values are inaccurately coded as calculated parameters; some laboratories had results that were measured in the units of a calculated parameter or had analyzed for a chemical species that was typically calculated, and put their data directly into the parameter codes defined as calculated. Therefore, data contained in calculated parameters is not always redundant. Hence, these calculated parameters were included as possible substitute values when combining multiple parameters.

Rules for combining multiple parameters developed here are similar to the rules developed by Goolsby and others (1999) with some additional substitutions. Water-quality parameters were combined using the first available concentration in the order listed for each constituent as follows. USGS parameter codes are included in parentheses (for example dissolved NH3 (00608)).

For Dissolved NH3, use:

  1. Dissolved NH3 (00608) in mg-N/L
  2. Total NH3 (00610) in mg-N/L

For Dissolved NO3 + NO2, use:

  1. Dissolved NO3 + NO2 (00631) in mg-N/L
  2. Total NO3 + NO2 (00630) in mg-N/L
  3. Dissolved NO3 (00618) in mg-N/L (calculated parameter: 00631 – 00613)
  4. Total NO3 (00620) in mg-N/L
  5. Dissolved NO3 (71851) in mg-NO3/L (calculated parameter: 00631 – 00613; convert from mg-NO3/L to mg-N/L by multiplying by 0.2259)
  6. Total NO3 (71850) in mg-NO3/L (convert from mg-NO3/L to mg-N/L by multiplying by 0.2259)

If using 3-6 above, then add NO2 from the sources listed below when available.

If both NO3 + NO2 are above or below the detection limit, add them. If only one is above the detection limit, use the value above detection limit. If NO2 is unavailable, use the value of NO3 only. (NO2 concentrations in surface waters are typically very low and often below the detection limit as NO2 can quickly convert to the more stable NO3 species. Therefore, it is reasonable to use dissolved NO3 concentrations as an approximation for dissolved NO3 + NO2.)

For TKN, use:

  1. TKN (00625) in mg-N/L
  2. Total N-Organic (00605) in mg-N/L + NH3 as defined above for dissolved NH3 (Both must be available and both must be above or below the detection limit—otherwise set to missing; parameter 00605 is a calculated value, calculated using the values of the following parameter codes as shown: 00625 – 00610, parameter code 00608 can be substituted for 00610 when calculating the value.)

For Dissolved OP, use:

  1. Dissolved (00671) OP in mg-P/L
  2. Dissolved OP (00660) in mg-PO4/L (calculated parameter: 00671; convert from mg-PO4/L to mg-P/L by multiplying by 0.3261)
  3. Total OP (70507) in mg-P/L
  4. Total OP (00650) in mg-PO4/L (calculated parameter: 70507; convert from mg-PO4/L to mg-P/L by multiplying by 0.3261)
  5. Dissolved OP (00653) in mg-PO4/L (convert from mg-PO4/L to mg-P/L by multiplying by 0.3261)

For Total P, use:

  1. Total P (00665) in mg-P/L
  2. Total P (71886) in mg-PO4/L (convert from mg-PO4/L to mg-P/L by multiplying by 0.3261)

5. Change From Laboratory Reporting Levels to Long-Term Method Detection Levels

Information on Long-Term Method Detection Levels (LT-MDL)

Starting on 10/1/98, the USGS NWQL changed their reporting level conventions for selected water-quality parameters (Foreman and others, 1998; Connor and others, 1999). In the past, values were censored in the NWIS QWDATA database to the minimum reporting level (MRL). The new conventions determine a long-term method detection level (LT-MDL) and a laboratory reporting level (LRL, typically twice the LT-MDL). When concentrations are below the LRL but above the LT-MDL, values are qualified as estimated. When concentrations are below the LT-MDL, values are set to less than the LRL (Oblinger Childress and others, 1999). To be consistent with the older data using the MRL convention and the required censoring conventions used by the load estimating software LOADEST (Runkel and others, 2004), estimated values were used as concentrations above the MRL and less than LRLs were changed to less than the LT-MDL.

For samples collected on or after 10/1/98, parameters with censored values were checked based on method code to see if the censored value was reported as the MRL or the LRL. If it was an LRL, the censored value was changed to the LT-MDL. A few censored parameters did not have method codes stored in the database, and the reporting level convention had to be determined on a case-by-case basis by considering the method code of recent samples collected at that station, the value of the censored concentration with respect to the concentrations of other samples, possible reporting levels used during that period, and the implementation schedule of the new reporting level conventions for that particular parameter.

6. Screen data for outliers based on statistical distributions

Means and standard deviations were calculated for each station and nutrient combination. For calculation purposes, values below the detection limit were set to one-half the detection limit. Concentrations of constituents vary in a log-normal pattern so the mean and standard deviations of each water-quality station and water-quality constituent combination is calculated from the natural log of the concentrations. If samples vary with a log-normal distribution, one would expect about 0.003 percent of samples to be 4 standard deviations below the mean and 0.003 percent of samples to be 4 standard deviations above the mean.

Screening water-quality constituent concentrations based on the mean ± 4 standard deviations resulted in about 0.31 percent of the values to be identified as outliers. Most outliers were low outliers, with 0.23 percent of the values identified as low outliers and 0.07 percent of the values identified as high outliers. Outliers were distributed fairly randomly across stations, constituents, and time. The ± 4 standard deviation screening level was used to remove outliers and appears to be reasonable because only a very small portion of concentrations are removed. Even if these outliers represent actual water-quality conditions, they represent a condition so unusual and short-lived that they should probably not be included for calibrating the flux estimation models. This is because the regression models are sensitive to gross outliers, due to the process of minimizing the sum of the squares, which gives the outliers undue weight in the model fitting, resulting in biased flux estimates.

7. Averaging multiple sample concentrations on same day

Because LOADEST was run on a daily time-step and because multiple samples collected on the same day typically represent replicate sampling, not different hydrologic conditions on the same day, concentrations from multiple samples were averaged into a single concentrations for that day. In cases for which concentrations to be averaged were both above and below the detection limit, the concentration of the first sample of the day was typically used.

8. Combining station data

Water-quality results for two stations on the Mississippi River, the one below Grafton, Illinois, and the one below Alton, Illinois, have been combined to yield a single long-term dataset. The sampling location was changed from Alton to Grafton in 1989 after testing demonstrated that data from these stations were comparable.


Connor, B.F., Foreman, W.T., and Maloney, Thomas, 1999, Announcement of NWQL reporting level changes: U.S. Geological Survey National Water Quality Laboratory Policy Memorandum 99.02, September 9, 1999.

Foreman, Bill, Connor, Brooke, Maloney, Tom, Zayhowski, Ed, and Vasquez, Juan, Reporting level changes for volatile organic compounds (Schedules 2020/2021), Inductively Coupled Plasma-Atomic Emission Spectrometry (ICP-AES), ammonia plus organic nitrogen and phosphorus (micro-Kjeldahl) in water methods at the National Water Quality Laboratory: U.S. Geological Survey National Water Quality Laboratory Technical Memorandum 98.07, September 28, 1998.

Goolsby, D.A., Battaglin, W.A., Lawrence, G.B., Artz, R.S., Aulenbach, B.T., Hooper, R.P., Keeney, D.R., and Stensland, G.J., 1999, Flux and sources of nutrients in the Mississippi-Atchafalaya River Basin—Topic 3, Report for the integrated assessment on hypoxia in the Gulf of Mexico: National Oceanic and Atmospheric Administration NOAA Coastal Ocean Program Decision Analysis Series No. 17, 129 p.

Oblinger Childress, C.J., Foreman, W.T., Connor, B.F., and Maloney, T.J., 1999, New reporting procedures based on long-term method detection levels and some considerations for interpretations of water-quality data provided by the U.S. Geological Survey National Water Quality Laboratory: U.S. Geological Survey Open-File Report 99-193, Reston, Virginia, 19 p.

Patton, C.J., 1997, Change in ammonia minimum reporting limit: U.S. Geological Survey National Water Quality Laboratory Technical Memorandum 97-10, June 20, 1997.

Rickert, D.A., 1992, Analytical methods—Discontinuation of the National Water Quality Laboratory determinations for “total” nitrate, “total” nitrite plus nitrate, “total” ammonia, and “total” orthophosphate (using the four-channel analyzer): U.S. Geological Survey Office of Water Quality Technical Memorandum 93.04, December 2, 1992.

Runkel, R.L., Crawford, C.G., and Cohn, T.A., 2004, Load estimator (LOADEST): A FORTRAN program for estimating constituent loads in streams and rivers: U.S. Geological Survey Techniques and Methods, book 4, chap. A5, 69 p.

U.S. Geological Survey, 1992, Programs and plans—Phosphorus methods and the quality of phosphorus data: U.S. Geological Survey Office of Water Quality Technical Memorandum 92.10, July 13, 1992.

U.S. Geological Survey, 2006, User's Manual for the National Water Information System of the U.S. Geological Survey—Water-Quality System, Version 4.6.: U.S. Geological Survey.

U.S. Geological Survey, variously dated, National field manual for the collection of water-quality data: U.S. Geological Survey Techniques of Water-Resources Investigations, book 9, chaps. A1-A9, available online at

Report Site Map > Sources and Preparation of Data Used for Nutrient Flux Estimation

[an error occurred while processing this directive]