Skip to topic | Skip to bottom
Home
Main
Main.QCNotesr1.14 - 05 Dec 2005 - 21:44 - JeremyCothrantopic end

Start of topic | Skip to actions

May 16, 2004

Earlier thoughts about encoding qa/qc into netCDF
http://carocoops.org/twiki/bin/view/Main/QcNotes

NDBC handbook on quality control
http://www.ndbc.noaa.gov/handbook.pdf

NDBC handbook update for realtime measurements
http://www.ndbc.noaa.gov/realtime.pdf

November 15, 2004

Jeremy,

Here is what Ed and I discussed so far for flag values in the netcdf files.

AQF (Aggregate Quality Flag): For entire series 0:best, 1:good, 2:fair, 3:questionable, 4: bad, -1:no tests done, -2: not applicable

STQF (Specific Test Quality Flag): Range test (within instrument/manufacturer range): R=yes; r=no Continuity test (within nearby points range): C=yes; c=no Georeference test: (within values expected for this location +/- a fudge factor): G=yes; g=no Online: (instrument is online and persumably collecting good data) O=yes; o=no (for example: tsg data where the pump is NOT flowing would have an "o" designation, or an instrument that was removed during the time series for repair or cal etc.);

Suggestions welcome for other tests.

The range and continuity tests would be for flagging wild points either from instrument malfunction or out - of - bound values in consecutive values. What I've always used as a simple test for continuity is one based on physical probability. For instance- a rate of change for water temperature could be used where a 4 degree change in ten minutes would not be physically feasible if the temperature returned quickly to the original time series average. Rate of change can be used for many scalar values as a way to flag wild points although some thought should be given to the physical processes that might be observed at a specific location (upwelling, eddies, current meanders etc). I also usually base my scalar editing on not only the rate of change - but also the duration of the change. Values which depart dramatically from the surrounding points for only one or two samples (at a high sample rate of say 10 or 15 minutes) usually are wild points. For vector values - such as currents and wind - it is probably better to use a standard deviation approach based on the entire time series or a normalized time series. In coastal areas, tidal currents can dramatically change direction and magnitude in a relatively short period of time - so a rate of change wouldn't work well for them.

For Georeference tests - again the physical processes that are being sampled will play a role in how large an area would be used for a "standard" value range in the region. Offshore, larger regions could be applied, however in areas of known frontal zones, tidal flows, and coastal zones affected more strongly on short time periods by wind, fresh water input or cold front passages, the regions may need to be smaller and based somewhat on perhaps water depth.

Nearby point comparisons are another possible test, however this may be better done at the SEACOOS aggregate level when all the data are available.

Liz

ooooooooooooooooooooooooooooooooooooooooooooooooooooooo

Sample of a netCDF standard with Quality flags (data is fake - so ncgen may not work)

Begin forwarded message:

netcdf sample_tsg_netcdf{

                 dimensions:
                         time = 302;
                         ndate = 19;
                 variables:
                 long time(time);
                         time: long_name = "originator time base in seconds";
                         time: standard_name = "time";
                         time: units = "sec since 1970-1-1 00:00:00 -0:00";
                         time: axis = "T";

                 char date_time_string(time,ndate);
                         date_time_string: long_name = "date_time string";
                         date_time_string: units = "string YYYY MM DD hh mm ss";
                         date_time_string: c_format = "%4d %2d %2d %2d %2d %2d";

                 float latitude(time);
                         latitude: long_name = "latitude in decimal degrees";
                         latitude: standard_name = "latitude";
                         latitude: units = "degrees_north" ;
                         latitude: dependency = "coordinate";
                         latitude: reference = "geographical coordinates" ;
                         latitude: axis = "Y";
                         latitude: valid_range = -90.,90.;

                 float longitude(time);
                         longitude: long_name = "longitude in decimal degrees";
                         longitude: standard_name = "longitude";
                         longitude: units = "degrees_east";
                         longitude: dependency = "coordinate";
                         longitude: reference = "geographical coordinates";
                         longitude: axis = "X";
                         longitude: valid_range = -180.,180.;

                 char Time(time,ndate);
                         Time: long_name: = "time character string";
                         Time: standard_name="time string";
                         Time: reference = "GMT";
                         Time: axis = "T";
                         Time: c_format = "%s:%s:%s";

                 float Conductivity(time);
                         Conductivity: long_name="conductivity of seawater";
                         Conductivity: standard_name="conductivity";
                         Conductivity: sensor_model="platinum-electrode glass cell";
                         Conductivity: sensor_serial_num="21003";
                         Conductivity: units="siemans per meter";
                         Conductivity: accuracy=".001";
                         Conductivity: resolution".0001";
                         Conductivity: valid_instrument_range=."0.,7";
                         Conductivity: geographic_range="0.,7";
                         Conductivity: dependency = "measurement";
                         Conductivity: reference="none";
                         Conductivity: sampling_interval="10";
                         Conductivity: averaging_interval="0";
                         Conductivity: physical_calibration="yes";
                         Conductivity: last_cal_date="21-Oct-2000";
                         Conductivity: aggregate_quality_level: "0";
                         Conductivity: axis="T";

                 char Conductivity_stqf(time);
                         Conductivity_stqf: long_name="specific test quality control flag for conductivity";
                         Conductivity_stqf:  
standard_name="conductivity_qcflag";
                         Conductivity_stqf: c_format = "%s:%s:%s:%s";

                 float Temperature(time);
                         Temperature: long_name="conductivity of seawater";
                         Temperature: standard_name="conductivity";
                         Temperature: sensor_model="thermistor";
                         Temperature: sensor_serial_num="21007";
                         Temperature: units="degrees Celsius";
                         Temperature: accuracy=".01";
                         Temperature: resolution=".001";
                         Temperature: dependency = "measurement";
                         Temperature: reference="none";
                         Temperature:  
valid_instrument_range=."-50.,50.";;
                         Temperature: geographic_range="15.,34.";
                         Temperature: sampling_interval="10";
                         Temperature: averaging_interval="0";
                         Temperature: physical_calibration="yes";
                         Temperature: last_cal_date="21-Oct-2000";
                         Temperature: axis="T";

                 char Temperature_stqf(time);
                         Temperature_stqf: long_name="specific test quality control flag for conductivity";
                         Temperature_stqf:  
standard_name="conductivity_qcflag";
                         Temperature_stqf: c_format = "%s:%s:%s:%s";

                 float Salinity(time);
                         Salinity: long_name="conductivity of seawater";
                         Salinity: standard_name="conductivity";
                         Salinity: units="PSU";
                         Salinity: accuracy=".01";
                         Salinity: resolution=".001";
                         Salinity: dependency = "derived";
                         Salinity: reference="none";
                         Salinity: valid_instrument_range=."0.,50.";;
                         Salinity: geographic_range="0.,40.";
                         Salinity: sampling_interval="10";
                         Salinity: averaging_interval="0";
                         Salinity: axis="T";


                 char Salinity_stqf(time);
                         Salinity_stqf: long_name="specific test quality control flag for conductivity";
                         Salinity_stqf:  
standard_name="conductivity_qcflag";
                         Salinity_stqf: c_format = "%s:%s:%s:%s";

                                 // global attributes:
                                 :title = "Explorer of the Seas Ocean Observations";
                                 :institution =  "University of Miami - RSMAS";
                                 :institution_code =  "um";
                                 :institution_url = "http://oceanlab.rsmas.miami.edu//";
                                 :institution_dods_url = "http://oceanlab.rsmas.miami.edu/RT/";
                                 :source = "surface observation";
                                 :reference = "http://www.rsmas.miami.edu/rccl/";
                                 :Conventions = "CF-1.0";

                                 :contact = "ewilliams@rsmas.miami.edu";
                                 :format = "Seacoos_RT";
                                 :instrument_type = "Explorer Instrument Suite";
                                 :data_type = "surface marine obs from moving platform";
                                 :instrument_code = "UM";
                                 :package_code = "TSG";
                                 : manufacturer_name="Seabird";
                                 : instrument_model="SBE-21 Seacat Thermosalinograph";
                                 : instr_serial_num= "3003";

                                 :platform_code = "ELWX5";

                                 :missing_value = -999;
                                 :FillValue = -999;
                                 :creation_date = "";
                                 :modification_date = "";
                                 :seacoos_rt_version = "";
                                 :castnet_url = "TBD, to describe data";

                                 :Latitude_max = 25.78421;
                                 :Latitude_min = 25.75743;
                                 :Longitude_max = -80.038314;
                                 :Longitude_min = -80.1834;
                                 :time_max = 1052560795;
                                 :time_min = 1052557202;
                                 :cruise_track = "";



data:

time =
1052557202,1052557214,1052557226,1052557237,1052557249,1052557261,105255
7273,1052557285,1052557297,1052557309,1052557321,1052557332,1052557344,1
052557356,1052557368,1052557379,1052557391,1052557403,1052557415,1052557
427,1052557439,1052557451,1052557463,1052557475,1052557487,1052557498,10
52557510,1052557522,1052557534,1052557546,1052557558,1052557569,10525575
81,1052557593,1052557605,1052557617,1052557629,1052557641,1052557653,105
2557665,1052557677,1052557689,1052557701,1052557713,1052557725,105255773
6,1052557748,1052557760,1052557772,1052557784,1052557796,1052557808,1052
557820,1052557832,1052557844,1052557856,1052557868,1052557880,1052557892
,1052557903,1052557915,1052557927,1052557939,1052557951,1052557963,10525
57975,1052557987,1052557999,1052558011,1052558023,1052558035,1052558047,
1052558059,1052558071,1052558083,1052558095,1052558107,1052558119,105255
8131,1052558143,1052558155,1052558167,1052558179,1052558191,1052558203,1
052558215,1052558227,1052558239,1052558251,1052558263,1052558275,1052558
287,1052558299,1052558311,1052558323,1052558335,1052558347,1052558359,10
52558371,1052558383,1052558395,1052558407,1052558419,1052558431,10525584
43,1052558455,1052558467,1052558479,1052558491,1052558503,1052558515,105
2558527,1052558538,1052558550,1052558562,1052558574,1052558586,105255859
8,1052558610,1052558622,1052558634,1052558646,1052558658,1052558670,1052
558682,1052558694,1052558706,1052558718,1052558730,1052558742,1052558754
,1052558765,1052558776,1052558788,1052558800,1052558812,1052558824,10525
58836,1052558848,1052558860,1052558872,1052558884,1052558896,1052558908,
1052558920,1052558931,1052558943,1052558955,1052558967,1052558979,105255
8991,1052559003,1052559015,1052559027,1052559039,1052559050,1052559062,1
052559074,1052559086,1052559098,1052559110,1052559122,1052559134,1052559
146,1052559158,1052559170,1052559182,1052559194,1052559206,1052559218,10
52559229,1052559241,1052559253,1052559265,1052559277,1052559289,10525593
01,1052559313,1052559325,1052559337,1052559349,1052559361,1052559373,105
2559384,1052559396,1052559408,1052559420,1052559432,1052559444,105255945
6,1052559468,1052559480,1052559492,1052559504,1052559516,1052559528,1052
559540,1052559552,1052559564,1052559576,1052559588,1052559600,1052559612
,1052559624,1052559636,1052559648,1052559660,1052559672,1052559684,10525
59696,1052559708,1052559720,1052559732,1052559744,1052559756,1052559768,
1052559780,1052559792,1052559804,1052559816,1052559828,1052559840,105255
9852,1052559864,1052559876,1052559888,1052559899,1052559912,1052559923,1
052559935,1052559948,1052559959,1052559971,1052559983,1052559995,1052560
007,1052560019,1052560031,1052560043,1052560055,1052560067,1052560079,10
52560091,1052560103,1052560115,1052560127,1052560139,1052560151,10525601
63,1052560174,1052560186,1052560198,1052560210,1052560222,1052560234,105
2560246,1052560258,1052560270,1052560282,1052560294,1052560306,105256031
8,1052560330,1052560342,1052560353,1052560365,1052560377,1052560389,1052
560401,1052560413,1052560425,1052560437,1052560449,1052560461,1052560473
,1052560485,1052560497,1052560509,1052560521,1052560533,1052560545,10525
60557,1052560569,1052560581,1052560593,1052560605,1052560617,1052560629,
1052560641,1052560653,1052560665,1052560677,1052560689,1052560700,105256
0712,1052560724,1052560736,1052560748,1052560760,1052560772,1052560784,1
052560795;


date_time_string = "2003 05 10 09 00 02","2003 05 10 09 00 14","2003 05 10 09 00 26","2003 05 10 09 00 37","2003 05 10 09 00 49","2003 05 10 09
01 01","2003 05 10 09 01 13","2003 05 10 09 01 25","2003 05 10 09 01
37","2003 05 10 09 01 49","2003 05 10 09 02 01","2003 05 10 09 02
12","2003 05 10 09 02 24","2003 05 10 09 02 36","2003 05 10 09 02
48","2003 05 10 09 02 59","2003 05 10 09 03 11","2003 05 10 09 03
23","2003 05 10 09 03 35","2003 05 10 09 03 47","2003 05 10 09 03
59","2003 05 10 09 04 11","2003 05 10 09 04 23","2003 05 10 09 04
35","2003 05 10 09 04 47","2003 05 10 09 04 58","2003 05 10 09 05
10","2003 05 10 09 05 22","2003 05 10 09 05 34","2003 05 10 09 05
46","2003 05 10 09 05 58","2003 05 10 09 06 09","2003 05 10 09 06
21","2003 05 10 09 06 33","2003 05 10 09 06 45","2003 05 10 09 06
57","2003 05 10 09 07 09","2003 05 10 09 07 21","2003 05 10 09 07
33","2003 05 10 09 07 45","2003 05 10 09 07 57","2003 05 10 09 08
09","2003 05 10 09 08 21","2003 05 10 09 08 33","2003 05 10 09 08
45","2003 05 10 09 08 56","2003 05 10 09 09 08","2003 05 10 09 09
20","2003 05 10 09 09 32","2003 05 10 09 09 44","2003 05 10 09 09
56","2003 05 10 09 10 08","2003 05 10 09 10 20","2003 05 10 09 10
32","2003 05 10 09 10 44","2003 05 10 09 10 56","2003 05 10 09 11
08","2003 05 10 09 11 20","2003 05 10 09 11 32","2003 05 10 09 11
43","2003 05 10 09 11 55","2003 05 10 09 12 07","2003 05 10 09 12
19","2003 05 10 09 12 31","2003 05 10 09 12 43","2003 05 10 09 12
55","2003 05 10 09 13 07","2003 05 10 09 13 19","2003 05 10 09 13
31","2003 05 10 09 13 43","2003 05 10 09 13 55","2003 05 10 09 14
07","2003 05 10 09 14 19","2003 05 10 09 14 31","2003 05 10 09 14
43","2003 05 10 09 14 55","2003 05 10 09 15 07","2003 05 10 09 15
19","2003 05 10 09 15 31","2003 05 10 09 15 43","2003 05 10 09 15
55","2003 05 10 09 16 07","2003 05 10 09 16 19","2003 05 10 09 16
31","2003 05 10 09 16 43","2003 05 10 09 16 55","2003 05 10 09 17
07","2003 05 10 09 17 19","2003 05 10 09 17 31","2003 05 10 09 17
43","2003 05 10 09 17 55","2003 05 10 09 18 07","2003 05 10 09 18
19","2003 05 10 09 18 31","2003 05 10 09 18 43","2003 05 10 09 18
55","2003 05 10 09 19 07","2003 05 10 09 19 19","2003 05 10 09 19
31","2003 05 10 09 19 43","2003 05 10 09 19 55","2003 05 10 09 20
07","2003 05 10 09 20 19","2003 05 10 09 20 31","2003 05 10 09 20
43","2003 05 10 09 20 55","2003 05 10 09 21 07","2003 05 10 09 21
19","2003 05 10 09 21 31","2003 05 10 09 21 43","2003 05 10 09 21
55","2003 05 10 09 22 07","2003 05 10 09 22 18","2003 05 10 09 22
30","2003 05 10 09 22 42","2003 05 10 09 22 54","2003 05 10 09 23
06","2003 05 10 09 23 18","2003 05 10 09 23 30","2003 05 10 09 23
42","2003 05 10 09 23 54","2003 05 10 09 24 06","2003 05 10 09 24
18","2003 05 10 09 24 30","2003 05 10 09 24 42","2003 05 10 09 24
54","2003 05 10 09 25 06","2003 05 10 09 25 18","2003 05 10 09 25
30","2003 05 10 09 25 42","2003 05 10 09 25 54","2003 05 10 09 26
05","2003 05 10 09 26 16","2003 05 10 09 26 28","2003 05 10 09 26
40","2003 05 10 09 26 52","2003 05 10 09 27 04","2003 05 10 09 27
16","2003 05 10 09 27 28","2003 05 10 09 27 40","2003 05 10 09 27
52","2003 05 10 09 28 04","2003 05 10 09 28 16","2003 05 10 09 28
28","2003 05 10 09 28 40","2003 05 10 09 28 51","2003 05 10 09 29
03","2003 05 10 09 29 15","2003 05 10 09 29 27","2003 05 10 09 29
39","2003 05 10 09 29 51","2003 05 10 09 30 03","2003 05 10 09 30
15","2003 05 10 09 30 27","2003 05 10 09 30 39","2003 05 10 09 30
50","2003 05 10 09 31 02","2003 05 10 09 31 14","2003 05 10 09 31
26","2003 05 10 09 31 38","2003 05 10 09 31 50","2003 05 10 09 32
02","2003 05 10 09 32 14","2003 05 10 09 32 26","2003 05 10 09 32
38","2003 05 10 09 32 50","2003 05 10 09 33 02","2003 05 10 09 33
14","2003 05 10 09 33 26","2003 05 10 09 33 38","2003 05 10 09 33
49","2003 05 10 09 34 01","2003 05 10 09 34 13","2003 05 10 09 34
25","2003 05 10 09 34 37","2003 05 10 09 34 49","2003 05 10 09 35
01","2003 05 10 09 35 13","2003 05 10 09 35 25","2003 05 10 09 35
37","2003 05 10 09 35 49","2003 05 10 09 36 01","2003 05 10 09 36
13","2003 05 10 09 36 24","2003 05 10 09 36 36","2003 05 10 09 36
48","2003 05 10 09 37 00","2003 05 10 09 37 12","2003 05 10 09 37
24","2003 05 10 09 37 36","2003 05 10 09 37 48","2003 05 10 09 38
00","2003 05 10 09 38 12","2003 05 10 09 38 24","2003 05 10 09 38
36","2003 05 10 09 38 48","2003 05 10 09 39 00","2003 05 10 09 39
12","2003 05 10 09 39 24","2003 05 10 09 39 36","2003 05 10 09 39
48","2003 05 10 09 40 00","2003 05 10 09 40 12","2003 05 10 09 40
24","2003 05 10 09 40 36","2003 05 10 09 40 48","2003 05 10 09 41
00","2003 05 10 09 41 12","2003 05 10 09 41 24","2003 05 10 09 41
36","2003 05 10 09 41 48","2003 05 10 09 42 00","2003 05 10 09 42
12","2003 05 10 09 42 24","2003 05 10 09 42 36","2003 05 10 09 42
48","2003 05 10 09 43 00","2003 05 10 09 43 12","2003 05 10 09 43
24","2003 05 10 09 43 36","2003 05 10 09 43 48","2003 05 10 09 44
00","2003 05 10 09 44 12","2003 05 10 09 44 24","2003 05 10 09 44
36","2003 05 10 09 44 48","2003 05 10 09 44 59","2003 05 10 09 45
12","2003 05 10 09 45 23","2003 05 10 09 45 35","2003 05 10 09 45
48","2003 05 10 09 45 59","2003 05 10 09 46 11","2003 05 10 09 46
23","2003 05 10 09 46 35","2003 05 10 09 46 47","2003 05 10 09 46
59","2003 05 10 09 47 11","2003 05 10 09 47 23","2003 05 10 09 47
35","2003 05 10 09 47 47","2003 05 10 09 47 59","2003 05 10 09 48
11","2003 05 10 09 48 23","2003 05 10 09 48 35","2003 05 10 09 48
47","2003 05 10 09 48 59","2003 05 10 09 49 11","2003 05 10 09 49
23","2003 05 10 09 49 34","2003 05 10 09 49 46","2003 05 10 09 49
58","2003 05 10 09 50 10","2003 05 10 09 50 22","2003 05 10 09 50
34","2003 05 10 09 50 46","2003 05 10 09 50 58","2003 05 10 09 51
10","2003 05 10 09 51 22","2003 05 10 09 51 34","2003 05 10 09 51
46","2003 05 10 09 51 58","2003 05 10 09 52 10","2003 05 10 09 52
22","2003 05 10 09 52 33","2003 05 10 09 52 45","2003 05 10 09 52
57","2003 05 10 09 53 09","2003 05 10 09 53 21","2003 05 10 09 53
33","2003 05 10 09 53 45","2003 05 10 09 53 57","2003 05 10 09 54
09","2003 05 10 09 54 21","2003 05 10 09 54 33","2003 05 10 09 54
45","2003 05 10 09 54 57","2003 05 10 09 55 09","2003 05 10 09 55
21","2003 05 10 09 55 33","2003 05 10 09 55 45","2003 05 10 09 55
57","2003 05 10 09 56 09","2003 05 10 09 56 21","2003 05 10 09 56
33","2003 05 10 09 56 45","2003 05 10 09 56 57","2003 05 10 09 57
09","2003 05 10 09 57 21","2003 05 10 09 57 33","2003 05 10 09 57
45","2003 05 10 09 57 57","2003 05 10 09 58 09","2003 05 10 09 58
20","2003 05 10 09 58 32","2003 05 10 09 58 44","2003 05 10 09 58
56","2003 05 10 09 59 08","2003 05 10 09 59 20","2003 05 10 09 59
32","2003 05 10 09 59 44","2003 05 10 09 59 55";


latitude =
25.780594,25.780422,25.780250,25.780090,25.779926,25.779762,25.779597,25
.779440,25.779280,25.779120,25.778962,25.778805,25.778640,25.778465,25.7
78293,25.778130,25.777952,25.777765,25.777568,25.777365,25.777152,25.776
940,25.776714,25.776490,25.776250,25.776030,25.775780,25.775540,25.77529
0,25.775047,25.774793,25.774570,25.774330,25.774090,25.773838,25.773580,
25.773340,25.773080,25.772828,25.772574,25.772318,25.772060,25.771796,25
.771540,25.771276,25.771042,25.770796,25.770550,25.770315,25.770082,25.7
69860,25.769634,25.769402,25.769175,25.768940,25.768716,25.768492,25.768
277,25.768054,25.767847,25.767634,25.767400,25.767160,25.766903,25.76663
2,25.766346,25.766058,25.765770,25.765490,25.765210,25.764930,25.764670,
25.764406,25.764160,25.763910,25.763680,25.763440,25.763195,25.762942,25
.762677,25.762412,25.762145,25.761868,25.761603,25.761333,25.761068,25.7
60810,25.760555,25.760297,25.760050,25.759800,25.759550,25.759314,25.759
090,25.758883,25.758670,25.758473,25.758280,25.758090,25.757912,25.75776
2,25.757655,25.757570,25.757498,25.757452,25.757430,25.757448,25.757506,
25.757600,25.757734,25.757896,25.758075,25.758273,25.758490,25.758740,25
.759000,25.759256,25.759510,25.759760,25.760000,25.760250,25.760490,25.7
60730,25.760958,25.761180,25.761400,25.761622,25.761843,25.762054,25.762
260,25.762467,25.762654,25.762850,25.763060,25.763253,25.763453,25.76365
5,25.763840,25.764032,25.764207,25.764403,25.764586,25.764770,25.764950,
25.765122,25.765286,25.765454,25.765626,25.765800,25.765968,25.766137,25
.766318,25.766486,25.766657,25.766835,25.766997,25.767170,25.767340,25.7
67512,25.767680,25.767852,25.768030,25.768205,25.768360,25.768540,25.768
722,25.768900,25.769078,25.769260,25.769430,25.769595,25.769773,25.76994
5,25.770118,25.770294,25.770458,25.770630,25.770800,25.770962,25.771134,
25.771294,25.771467,25.771628,25.771784,25.771945,25.772100,25.772258,25
.772418,25.772572,25.772730,25.772894,25.773040,25.773205,25.773363,25.7
73516,25.773664,25.773818,25.773972,25.774128,25.774282,25.774425,25.774
593,25.774747,25.774903,25.775058,25.775200,25.775357,25.775510,25.77566
4,25.775818,25.775972,25.776124,25.776270,25.776422,25.776570,25.776718,
25.776863,25.777016,25.777164,25.777308,25.777442,25.777598,25.777723,25
.777862,25.778008,25.778150,25.778270,25.778411,25.778540,25.778670,25.7
78820,25.778950,25.779080,25.779224,25.779368,25.779503,25.779630,25.779
770,25.779900,25.780048,25.780177,25.780310,25.780440,25.780560,25.78070
6,25.780830,25.780950,25.781040,25.781160,25.781265,25.781388,25.781495,
25.781612,25.781730,25.781850,25.781954,25.782060,25.782170,25.782273,25
.782360,25.782473,25.782580,25.782698,25.782780,25.782878,25.782974,25.7
83070,25.783177,25.783265,25.783353,25.783440,25.783520,25.783600,25.783
670,25.783753,25.783840,25.783913,25.783995,25.784045,25.784105,25.78413
8,25.784168,25.784195,25.784210,25.784205,25.784167,25.784133,25.784074,
25.784014,25.783953,25.783886,25.783820,25.783760,25.783705,25.783642,25
.783580,25.783522,25.783456,25.783395,25.783324,25.783265,25.783210;


longitude =
-80.038314,-80.038876,-80.039430,-80.039940,-80.040486,-80.041034,-
80.041570,-80.042100,-80.042640,-80.043170,-80.043705,-80.044205,-
80.044750,-80.045295,-80.045843,-80.046350,-80.046908,-80.047477,-
80.048052,-80.048640,-80.049250,-80.049870,-80.050514,-80.051160,-
80.051815,-80.052427,-80.053106,-80.053792,-80.054486,-80.055200,-
80.055918,-80.056582,-80.057320,-80.058064,-80.058812,-80.059560,-
80.060320,-80.061072,-80.061834,-80.062602,-80.063364,-80.064130,-
80.064898,-80.065666,-80.066448,-80.067156,-80.067942,-80.068734,-
80.069540,-80.070348,-80.071154,-80.071966,-80.072778,-80.073595,-
80.074410,-80.075224,-80.076048,-80.076872,-80.077696,-80.078453,-
80.079286,-80.080110,-80.080928,-80.081733,-80.082536,-80.083334,-
80.084126,-80.084920,-80.085725,-80.086530,-80.087345,-80.088160,-
80.088976,-80.089794,-80.090600,-80.091416,-80.092220,-80.093030,-
80.093832,-80.094640,-80.095452,-80.096265,-80.097074,-80.097883,-
80.098677,-80.099466,-80.100246,-80.101015,-80.101777,-80.102528,-
80.103284,-80.104020,-80.104764,-80.105510,-80.106263,-80.107010,-
80.107753,-80.108500,-80.109250,-80.110004,-80.110766,-80.111530,-
80.112300,-80.113068,-80.113846,-80.114618,-80.115404,-80.116168,-
80.116920,-80.117654,-80.118380,-80.119085,-80.119723,-80.120406,-
80.121074,-80.121734,-80.122382,-80.123024,-80.123657,-80.124290,-
80.124905,-80.125510,-80.126110,-80.126718,-80.127293,-80.127868,-
80.128434,-80.129000,-80.129552,-80.130100,-80.130635,-80.131122,-
80.131650,-80.132166,-80.132663,-80.133167,-80.133650,-80.134130,-
80.134606,-80.135047,-80.135537,-80.136002,-80.136450,-80.136890,-
80.137332,-80.137734,-80.138166,-80.138598,-80.139030,-80.139458,-
80.139883,-80.140312,-80.140736,-80.141157,-80.141595,-80.141990,-
80.142420,-80.142850,-80.143278,-80.143710,-80.144138,-80.144570,-
80.144995,-80.145415,-80.145840,-80.146258,-80.146670,-80.147082,-
80.147490,-80.147897,-80.148257,-80.148659,-80.149047,-80.149434,-
80.149818,-80.150208,-80.150590,-80.150975,-80.151373,-80.151758,-
80.152154,-80.152547,-80.152924,-80.153286,-80.153675,-80.154050,-
80.154424,-80.154800,-80.155173,-80.155550,-80.155920,-80.156290,-
80.156665,-80.157040,-80.157410,-80.157790,-80.158168,-80.158542,-
80.158924,-80.159300,-80.159660,-80.160060,-80.160427,-80.160800,-
80.161187,-80.161570,-80.161943,-80.162320,-80.162700,-80.163078,-
80.163460,-80.163824,-80.164200,-80.164570,-80.164940,-80.165282,-
80.165650,-80.166002,-80.166360,-80.166720,-80.167060,-80.167412,-
80.167740,-80.168108,-80.168450,-80.168790,-80.169105,-80.169469,-
80.169780,-80.170105,-80.170468,-80.170770,-80.171102,-80.171426,-
80.171762,-80.172093,-80.172403,-80.172750,-80.173048,-80.173404,-
80.173723,-80.174037,-80.174345,-80.174616,-80.174938,-80.175230,-
80.175512,-80.175780,-80.176064,-80.176310,-80.176603,-80.176865,-
80.177138,-80.177433,-80.177710,-80.177976,-80.178230,-80.178484,-
80.178720,-80.178962,-80.179223,-80.179460,-80.179715,-80.179940,-
80.180148,-80.180370,-80.180600,-80.180828,-80.181055,-80.181263,-
80.181480,-80.181687,-80.181883,-80.182070,-80.182265,-80.182430,-
80.182577,-80.182712,-80.182840,-80.182952,-80.183056,-80.183143,-
80.183200,-80.183250,-80.183300,-80.183347,-80.183378,-80.183400,-
80.183400,-80.183387,-80.183362,-80.183340,-80.183325,-80.183298,-
80.183268,-80.183240,-80.183204,-80.183176,-80.183135,-80.183086,-
80.183035,-80.182980;


Time =
"09:00:02","09:00:14","09:00:26","09:00:37","09:00:49","09:01:01","09:01
:13","09:01:25","09:01:37","09:01:49","09:02:01","09:02:12","09:02:24"," 
09:02:36","09:02:48","09:02:59","09:03:11","09:03:23","09:03:35","09:03: 
47","09:03:59","09:04:11","09:04:23","09:04:35","09:04:47","09:04:58","0
9:05:10","09:05:22","09:05:34","09:05:46","09:05:58","09:06:09","09:06:2
1","09:06:33","09:06:45","09:06:57","09:07:09","09:07:21","09:07:33","09
:07:45","09:07:57","09:08:09","09:08:21","09:08:33","09:08:45","09:08:56
","09:09:08","09:09:20","09:09:32","09:09:44","09:09:56","09:10:08","09: 
10:20","09:10:32","09:10:44","09:10:56","09:11:08","09:11:20","09:11:32" 
,"09:11:43","09:11:55","09:12:07","09:12:19","09:12:31","09:12:43","09:1
2:55","09:13:07","09:13:19","09:13:31","09:13:43","09:13:55","09:14:07",
"09:14:19","09:14:31","09:14:43","09:14:55","09:15:07","09:15:19","09:15
:31","09:15:43","09:15:55","09:16:07","09:16:19","09:16:31","09:16:43"," 
09:16:55","09:17:07","09:17:19","09:17:31","09:17:43","09:17:55","09:18: 
07","09:18:19","09:18:31","09:18:43","09:18:55","09:19:07","09:19:19","0
9:19:31","09:19:43","09:19:55","09:20:07","09:20:19","09:20:31","09:20:4
3","09:20:55","09:21:07","09:21:19","09:21:31","09:21:43","09:21:55","09
:22:07","09:22:18","09:22:30","09:22:42","09:22:54","09:23:06","09:23:18
","09:23:30","09:23:42","09:23:54","09:24:06","09:24:18","09:24:30","09: 
24:42","09:24:54","09:25:06","09:25:18","09:25:30","09:25:42","09:25:54" 
,"09:26:05","09:26:16","09:26:28","09:26:40","09:26:52","09:27:04","09:2
7:16","09:27:28","09:27:40","09:27:52","09:28:04","09:28:16","09:28:28",
"09:28:40","09:28:51","09:29:03","09:29:15","09:29:27","09:29:39","09:29
:51","09:30:03","09:30:15","09:30:27","09:30:39","09:30:50","09:31:02"," 
09:31:14","09:31:26","09:31:38","09:31:50","09:32:02","09:32:14","09:32: 
26","09:32:38","09:32:50","09:33:02","09:33:14","09:33:26","09:33:38","0
9:33:49","09:34:01","09:34:13","09:34:25","09:34:37","09:34:49","09:35:0
1","09:35:13","09:35:25","09:35:37","09:35:49","09:36:01","09:36:13","09
:36:24","09:36:36","09:36:48","09:37:00","09:37:12","09:37:24","09:37:36
","09:37:48","09:38:00","09:38:12","09:38:24","09:38:36","09:38:48","09: 
39:00","09:39:12","09:39:24","09:39:36","09:39:48","09:40:00","09:40:12" 
,"09:40:24","09:40:36","09:40:48","09:41:00","09:41:12","09:41:24","09:4
1:36","09:41:48","09:42:00","09:42:12","09:42:24","09:42:36","09:42:48",
"09:43:00","09:43:12","09:43:24","09:43:36","09:43:48","09:44:00","09:44
:12","09:44:24","09:44:36","09:44:48","09:44:59","09:45:12","09:45:23"," 
09:45:35","09:45:48","09:45:59","09:46:11","09:46:23","09:46:35","09:46: 
47","09:46:59","09:47:11","09:47:23","09:47:35","09:47:47","09:47:59","0
9:48:11","09:48:23","09:48:35","09:48:47","09:48:59","09:49:11","09:49:2
3","09:49:34","09:49:46","09:49:58","09:50:10","09:50:22","09:50:34","09
:50:46","09:50:58","09:51:10","09:51:22","09:51:34","09:51:46","09:51:58
","09:52:10","09:52:22","09:52:33","09:52:45","09:52:57","09:53:09","09: 
53:21","09:53:33","09:53:45","09:53:57","09:54:09","09:54:21","09:54:33" 
,"09:54:45","09:54:57","09:55:09","09:55:21","09:55:33","09:55:45","09:5
5:57","09:56:09","09:56:21","09:56:33","09:56:45","09:56:57","09:57:09",
"09:57:21","09:57:33","09:57:45","09:57:57","09:58:09","09:58:20","09:58
:32","09:58:44","09:58:56","09:59:08","09:59:20","09:59:32","09:59:44"," 
09:59:55";


Conductivity =
5.7817,5.7825,5.7831,5.7839,5.7846,5.7853,5.7861,5.7867,5.7875,5.7881,5. 
7887,5.7895,5.7904,5.7910,5.7916,5.7921,5.7928,5.7936,5.7944,5.7948,5.79
58,5.7961,5.7969,5.7977,5.7984,5.7991,5.7997,5.8007,5.8012,5.8019,5.8027
,5.8032,5.8040,5.8045,5.8052,5.8060,5.8068,5.8072,5.8080,5.8087,5.8095,5
.8103,5.8109,5.8114,5.8123,5.8130,5.8136,5.8143,5.8151,5.8159,5.8165,5.8
176,5.8181,5.8186,5.8194,5.8203,5.8210,5.8217,5.8225,5.8232,5.8241,5.824
6,5.8254,5.8262,5.8268,5.8277,5.8286,5.8294,5.8301,5.8309,5.8316,5.8324,
5.8332,5.8341,5.8349,5.8358,5.8367,5.8375,5.8383,5.8392,5.8401,5.8412,5. 
8421,5.8430,5.8439,5.8448,5.8458,5.8468,5.8478,5.8489,5.8500,5.8509,5.85
21,5.8531,5.8543,5.8556,5.8565,5.8579,5.8592,5.8603,5.8616,5.8629,5.8642
,5.8656,5.8669,5.8683,5.8697,5.8710,5.8727,5.8740,5.8757,5.8773,5.8789,5
.8805,5.8820,5.8837,5.8855,5.8871,5.8888,5.8909,5.8925,5.8945,5.8961,5.8
981,5.9000,5.9019,5.9041,5.9060,5.9082,5.9101,5.9123,5.9144,5.9166,5.918
9,5.9209,5.9232,5.9255,5.9277,5.9301,5.9324,5.9347,5.9368,5.9393,5.9415,
5.9438,5.9463,5.9487,5.9509,5.9532,5.9555,5.9578,5.9603,5.9626,5.9651,5. 
9675,5.9698,5.9723,5.9747,5.9769,5.9792,5.9816,5.9841,5.9864,5.9889,5.99
13,5.9938,5.9961,5.9985,6.0010,6.0032,6.0057,6.0079,6.0105,6.0126,6.0151
,6.0173,6.0195,6.0218,6.0240,6.0265,6.0287,6.0311,6.0332,6.0357,6.0377,6
.0399,6.0421,6.0442,6.0464,6.0486,6.0509,6.0530,6.0552,6.0572,6.0593,6.0
615,6.0636,6.0654,6.0676,6.0696,6.0716,6.0736,6.0757,6.0777,6.0796,6.081
5,6.0834,6.0854,6.0870,6.0891,6.0909,6.0928,6.0946,6.0965,6.0981,6.1000,
6.1015,6.1033,6.1052,6.1069,6.1085,6.1103,6.1116,6.1133,6.1150,6.1167,6. 
1180,6.1194,6.1210,6.1226,6.1241,6.1257,6.1271,6.1285,6.1301,6.1315,6.13
28,6.1343,6.1356,6.1370,6.1382,6.1397,6.1409,6.1424,6.1437,6.1448,6.1460
,6.1473,6.1486,6.1498,6.1511,6.1523,6.1534,6.1547,6.1557,6.1568,6.1579,6
.1590,6.1601,6.1612,6.1624,6.1634,6.1646,6.1655,6.1666,6.1676,6.1684,6.1
695,6.1704,6.1714,6.1726,6.1733,6.1743,6.1755,6.1762,6.1772,6.1778,6.179
0,6.1796,6.1806,6.1815,6.1824,6.1831,6.1841,6.1850,6.1857,6.1866,6.1876,
6.1882,6.1890,6.1898,6.1907,6.1913,6.1921,6.1927,6.1934,6.1943,6.1951,6. 
1957,6.1965,6.1972,6.1979;

Temperature =
27.7350,27.7393,27.7443,27.7493,27.7537,27.7587,27.7643,27.7687,27.7737,
27.7780,27.7830,27.7880,27.7924,27.7974,27.8024,27.8068,27.8117,27.8167,
27.8217,27.8261,27.8305,27.8367,27.8423,27.8460,27.8517,27.8560,27.8616,
27.8666,27.8716,27.8766,27.8816,27.8866,27.8922,27.8959,27.9028,27.9065,
27.9134,27.9177,27.9233,27.9283,27.9333,27.9383,27.9445,27.9489,27.9545,
27.9601,27.9663,27.9707,27.9763,27.9819,27.9881,27.9918,27.9993,28.0043,
28.0099,28.0155,28.0217,28.0260,28.0335,28.0385,28.0447,28.0503,28.0540,
28.0634,28.0665,28.0739,28.0801,28.0864,28.0932,28.0982,28.1056,28.1112,
28.1205,28.1237,28.1342,28.1392,28.1466,28.1528,28.1597,28.1684,28.1721,
28.1820,28.1895,28.1957,28.2050,28.2112,28.2199,28.2267,28.2354,28.2428,
28.2509,28.2584,28.2677,28.2757,28.2832,28.2918,28.3011,28.3098,28.3179,
28.3272,28.3358,28.3439,28.3526,28.3612,28.3699,28.3792,28.3879,28.3971,
28.4058,28.4139,28.4213,28.4306,28.4386,28.4485,28.4559,28.4646,28.4733,
28.4813,28.4906,28.4986,28.5073,28.5153,28.5233,28.5332,28.5413,28.5493,
28.5586,28.5684,28.5771,28.5851,28.5938,28.6030,28.6123,28.6203,28.6302,
28.6388,28.6475,28.6567,28.6647,28.6746,28.6832,28.6919,28.7017,28.7110,
28.7190,28.7289,28.7381,28.7474,28.7566,28.7658,28.7751,28.7837,28.7936,
28.8034,28.8133,28.8219,28.8323,28.8416,28.8508,28.8600,28.8717,28.8810,
28.8908,28.9013,28.9123,28.9215,28.9338,28.9424,28.9560,28.9640,28.9756,
28.9867,28.9965,29.0100,29.0192,29.0315,29.0432,29.0530,29.0659,29.0775,
29.0904,29.1021,29.1162,29.1260,29.1432,29.1511,29.1689,29.1806,29.1934,
29.2094,29.2210,29.2375,29.2504,29.2657,29.2798,29.2944,29.3097,29.3263,
29.3391,29.3580,29.3690,29.3898,29.4008,29.4210,29.4344,29.4527,29.4674,
29.4851,29.5010,29.5193,29.5339,29.5534,29.5681,29.5863,29.6040,29.6192,
29.6387,29.6521,29.6716,29.6868,29.7045,29.7234,29.7374,29.7574,29.7720,
29.7896,29.8055,29.8213,29.8389,29.8534,29.8741,29.8875,29.9069,29.9208,
29.9384,29.9542,29.9712,29.9857,30.0039,30.0179,30.0355,30.0500,30.0663,
30.0821,30.0978,30.1124,30.1281,30.1438,30.1584,30.1723,30.1874,30.2019,
30.2158,30.2315,30.2442,30.2599,30.2738,30.2871,30.3022,30.3143,30.3288,
30.3421,30.3553,30.3680,30.3819,30.3946,30.4078,30.4205,30.4325,30.4470,
30.4573,30.4723,30.4795,30.5000,30.5054,30.5235,30.5313,30.5416,30.5566,
30.5602,30.5825,30.5837,30.6024,30.6096,30.6192,30.6354,30.6384,30.6583,
30.6637,30.6775,30.6865,30.6955,30.7063,30.7136,30.7250,30.7346,30.7430,
30.7556,30.7634,30.7754,30.7832,30.7940;

Salinity =
36.3266,36.3288,36.3292,36.3310,36.3325,36.3336,36.3349,36.3357,36.3375,
36.3383,36.3394,36.3412,36.3441,36.3445,36.3448,36.3450,36.3460,36.3478,
36.3496,36.3490,36.3527,36.3500,36.3512,36.3540,36.3546,36.3561,36.3560,
36.3592,36.3588,36.3599,36.3617,36.3613,36.3626,36.3632,36.3628,36.3656,
36.3659,36.3660,36.3673,36.3684,36.3702,36.3719,36.3713,36.3715,36.3734,
36.3740,36.3734,36.3750,36.3763,36.3775,36.3769,36.3818,36.3795,36.3792,
36.3805,36.3824,36.3826,36.3841,36.3839,36.3850,36.3865,36.3857,36.3884,
36.3875,36.3893,36.3899,36.3914,36.3922,36.3918,36.3936,36.3927,36.3940,
36.3924,36.3963,36.3938,36.3962,36.3968,36.3976,36.3979,36.3975,36.4009,
36.4010,36.4015,36.4030,36.4021,36.4043,36.4046,36.4063,36.4066,36.4086,
36.4101,36.4106,36.4118,36.4126,36.4152,36.4176,36.4167,36.4198,36.4227,
36.4232,36.4256,36.4285,36.4315,36.4346,36.4370,36.4396,36.4427,36.4446,
36.4498,36.4527,36.4588,36.4628,36.4677,36.4713,36.4760,36.4819,36.4877,
36.4927,36.4974,36.5058,36.5103,36.5180,36.5229,36.5292,36.5362,36.5440,
36.5521,36.5577,36.5664,36.5734,36.5820,36.5895,36.5976,36.6074,36.6144,
36.6237,36.6330,36.6411,36.6516,36.6600,36.6693,36.6772,36.6869,36.6957,
36.7055,36.7153,36.7248,36.7329,36.7417,36.7505,36.7593,36.7707,36.7790,
36.7887,36.7977,36.8070,36.8162,36.8257,36.8338,36.8433,36.8509,36.8610,
36.8694,36.8786,36.8866,36.8968,36.9032,36.9131,36.9206,36.9297,36.9379,
36.9445,36.9549,36.9589,36.9691,36.9748,36.9816,36.9899,36.9951,37.0033,
37.0085,37.0161,37.0196,37.0293,37.0298,37.0394,37.0408,37.0463,37.0515,
37.0543,37.0611,37.0628,37.0680,37.0699,37.0741,37.0779,37.0805,37.0801,
37.0853,37.0843,37.0896,37.0872,37.0931,37.0913,37.0946,37.0935,37.0952,
37.0952,37.0939,37.0942,37.0952,37.0931,37.0942,37.0931,37.0911,37.0923,
37.0875,37.0895,37.0875,37.0874,37.0847,37.0824,37.0805,37.0767,37.0771,
37.0751,37.0725,37.0699,37.0672,37.0669,37.0613,37.0619,37.0565,37.0553,
37.0527,37.0501,37.0459,37.0450,37.0399,37.0387,37.0340,37.0331,37.0287,
37.0268,37.0236,37.0199,37.0160,37.0128,37.0105,37.0080,37.0053,37.0023,
36.9991,36.9959,36.9930,36.9885,36.9860,36.9833,36.9792,36.9774,36.9745,
36.9712,36.9691,36.9656,36.9624,36.9595,36.9548,36.9526,36.9495,36.9453,
36.9456,36.9388,36.9401,36.9326,36.9332,36.9262,36.9250,36.9253,36.9179,
36.9219,36.9111,36.9162,36.9068,36.9081,36.9068,36.8992,36.9030,36.8947,
36.8946,36.8895,36.8881,36.8873,36.8831,36.8831,36.8785,36.8759,36.8756,
36.8714,36.8695,36.8658,36.8646,36.8612;

         }
oooooooooooooooooooooooooooooooooooooooooooooooooooo

my comments from earlier http://carocoops.org/twiki_dmcc/bin/view/Main/JCNotes#Seacoos_netCDF_convention

-- JeremyCothran - 10 Dec 2004

January 16, 2005

Qartod excerpts

I've excerpted the following sections of the Qartod I final report which I think are reflected in the suggested netCDF attribute scheme.

=============================================================

From the QARTOD I Workshop December, 2003
final report document http://nautilus.baruch.sc.edu/twiki/pub/Main/WebHome/QARTOD_final_09.pdf

3. The real-time quality is best described by an aggregate quality flag (a simple overall descriptor) and a detailed quality test record (indicative of the results of any individual quality tests applied) to suit both the common user and the real-time scientist. The aggregate quality levels were recommended to be few (although the order is arbitrary):

-9 = missing value
0 = quality not evaluated
1 = bad
2 = questionable/suspect
3 = good

4. The quality flags and quality test descriptions must be sufficiently described in the accompanying metadata.

The group recommended that a series of automated tests be performed on the real-time data. These tests include

Transmission/validity check

  • Valid message received (e.g. checksum validation)
  • Time stamp validity
  • Check for service schedule of hardware (Field engineers must communicate with data/QC analysts)
Gross error limit checks
  • Measurement exceeds geophysically possible ranges
Threshold checks Measurement exceeds geophysically plausible range
  • Tied to expected local conditions
Rate of change consistency check
  • Is the temporal variability within expected limits?
  • First derivative, Second derivative…
Comparisons to other data sources
  • Climatology
    • Seasonal, weekly means and measured variability
    • Must specify climatology used in metadata
  • Comparisons to nearest neighbor
    • Compare to magnitude at other nearby stations
    • Compute gradients between stations
  • Comparison to co-located redundant sensor(s)
  • Cross-Variable tests using other measured parameters
Use temperature-salinity relationship
  • density profile stability
Other ‘sanity’ checks based on known physical/biological/chemical limitations
Comparison vs. model products
  • Sanity checks with respect to models, requires rapid access to modeling products and an independent model to avoid conflicts
  • Specify models used in metadata
Statistical tests
  • Random error of sample (Standard deviation/Root Mean Square)
  • Spectral, modal, “spikiness” tests
Artificial Intelligence (AI), neural network tests
Other tests will be defined as the user requires; such tests to be explained in the metadata

=============================================================

Sample qc netcdf full

netcdf sample_qc_netcdf {

 dimensions:
    time = 50;
    ndate = 19;
 variables:
 long time(time);
    time: long_name = "originator time base in seconds";
    time: standard_name = "time";
    time: units = "sec since 1970-1-1 00:00:00 -0:00";
    time: axis = "T";

 char date_time_string(time,ndate);
    date_time_string: long_name = "date_time string";
    date_time_string: units = "string YYYY MM DD hh mm ss";
    date_time_string: c_format = "%4d %2d %2d %2d %2d %2d";

 float latitude(time);
    latitude: long_name = "latitude in decimal degrees";
    latitude: standard_name = "latitude";
    latitude: units = "degrees_north" ;
    latitude: dependency = "coordinate";
    latitude: reference = "geographical coordinates" ;
    latitude: axis = "Y";
    latitude: valid_range = -90.,90.;

 float longitude(time);
    longitude: long_name = "longitude in decimal degrees";
    longitude: standard_name = "longitude";
    longitude: units = "degrees_east";
    longitude: dependency = "coordinate";
    longitude: reference = "geographical coordinates";
    longitude: axis = "X";
    longitude: valid_range = -180.,180.;

 float conductivity(time);
    conductivity: long_name="conductivity of seawater";
    conductivity: standard_name="conductivity";
    conductivity: units="siemans per meter";

 char conductivity_qcflag(time);
    conductivity_qcflag: long_name="specific test quality control flag for conductivity";
    conductivity_qcflag: standard_name="conductivity_qcflag";
    conductivity_qcflag: c_format = "%s:%s:%s:%s";
    conductivity_qcflag: aggregate_quality_level: "3";
    conductivity_qcflag: qc_documentation_url: "http://www.qartod/handbook.pdf, http://www.mysite/myimplementation.html"; 

    conductivity_qcflag: sensor_model="platinum-electrode glass cell";
    conductivity_qcflag: sensor_serial_num="21003";
    conductivity_qcflag: accuracy=".001";
    conductivity_qcflag: resolution".0001";
    conductivity_qcflag: valid_instrument_range=."0.,7.";
    conductivity_qcflag: max_continuous_change="4.";
    conductivity_qcflag: geographic_range="0.,7.";
    conductivity_qcflag: dependency = "measurement";
    conductivity_qcflag: reference="none";
    conductivity_qcflag: sampling_interval="10";
    conductivity_qcflag: averaging_interval="0";
    conductivity_qcflag: physical_calibration="yes";
    conductivity_qcflag: last_cal_date="21-Oct-2000";

 float temperature(time);
    temperature: long_name="temperature of seawater";
    temperature: standard_name="temperature";
    temperature: units="degrees Celsius";

 char temperature_qcflag(time);
    temperature_qcflag: long_name="specific test quality control flag for temperature";
    temperature_qcflag: standard_name="temperature_qcflag";
    temperature_qcflag: c_format = "%s:%s:%s:%s";

    temperature_qcflag: sensor_model="thermistor";
    temperature_qcflag: sensor_serial_num="21007";
    temperature_qcflag: accuracy=".01";
    temperature_qcflag: resolution=".001";
    temperature_qcflag: dependency = "measurement";
    temperature_qcflag: reference="none";
    temperature_qcflag: valid_instrument_range=."-50.,50.";
    temperature_qcflag: geographic_range="15.,34.";
    temperature_qcflag: sampling_interval="10";
    temperature_qcflag: averaging_interval="0";
    temperature_qcflag: physical_calibration="yes";
    temperature_qcflag: last_cal_date="21-Oct-2000";

 float salinity(time);
    salinity: long_name="salinity of seawater";
    salinity: standard_name="salinity";
    salinity: units="PSU";

 char salinity_qcflag(time);
    salinity_qcflag: long_name="specific test quality control flag for salinity";
    salinity_qcflag: standard_name="salinity_qcflag";
    salinity_qcflag: c_format = "%s:%s:%s:%s";
    salinity_qcflag: aggregate_quality_level: "3";
    salinity_qcflag: qc_documentation_url: "http://www.qartod/handbook.pdf, http://www.mysite/myimplementation.html"; 

    salinity_qcflag: accuracy=".01";
    salinity_qcflag: resolution=".001";
    salinity_qcflag: dependency = "derived";
    salinity_qcflag: reference="none";
    salinity_qcflag: valid_instrument_range=."0.,50.";;
    salinity_qcflag: geographic_range="0.,40.";
    salinity_qcflag: sampling_interval="10";
    salinity_qcflag: averaging_interval="0";
    salinity_qcflag: aggregate_quality_level: "3";    

 // global attributes
 :title = "Explorer of the Seas Ocean Observations";
 :institution =  "University of Miami - RSMAS";
 :institution_code =  "um";
 :institution_url = "http://oceanlab.rsmas.miami.edu//";
 :institution_dods_url = "http://oceanlab.rsmas.miami.edu/RT/";
 :source = "surface observation";
 :reference = "http://www.rsmas.miami.edu/rccl/";
 :conventions = "CF-1.0, SEACOOS-NETCDF-2.0, SEACOOS-QC-1.0";

 :contact = "ewilliams@rsmas.miami.edu";
 :format = "Seacoos_RT";
 :instrument_type = "Explorer Instrument Suite";
 :data_type = "surface marine obs from moving platform";
 :instrument_code = "UM";
 :package_code = "TSG";
 :manufacturer_name="Seabird";
 :instrument_model="SBE-21 Seacat Thermosalinograph";
 :instr_serial_num= "3003";

 :platform_code = "ELWX5";

 :missing_value = -999;
 :FillValue = -999;
 :creation_date = "";
 :modification_date = "";
 :seacoos_rt_version = "";
 :castnet_url = "TBD, to describe data";

 :Latitude_max = 25.78421;
 :Latitude_min = 25.75743;
 :Longitude_max = -80.038314;
 :Longitude_min = -80.1834;
 :time_max = 1052560795;
 :time_min = 1052557202;
 :cruise_track = "";

 // global variable attribute defaults - variable attribute defaulted to the below if not defined within the variable

 // aggregate_quality_level values
 // -9 = missing value
 // 0 = quality not evaluated
 // 1 = bad
 // 2 = questionable/suspect
 // 3 = good

 :aggregate_quality_level: "3";
 :qc_documentation_url: "http://www.qartod/handbook.pdf, http://www.mysite/myimplementation.html"; 

data:

time = 1052557202,1052557214,1052557226, ...

date_time_string = "2003 05 10 09 00 02","2003 05 10 09 00 14","2003 05 10 09 00 26", ...

latitude = 25.780594,25.780422,25.780250, ...

longitude = -80.038314,-80.038876,-80.039430, ...

conductivity = 5.7817,5.7825,5.7831, ...

conductivity_qcflag = "O:R:C:G", "O:r:c:g", "o:x:x:x", "O:R:C:g 1", "O:R:C:G", ...

temperature = 27.7350,27.7393,27.7443, ...

temperature_qcflag = "O:R:C:G", "O:R:C:G", "O:R:C:G", ...

salinity = 36.3266,36.3288,36.3292, ...

salinity_qcflag = "O:R:C:G", "O:R:C:G", "O:R:C:G", ...

}

Sample qc netcdf short

netcdf sample_qc_netcdf {

 dimensions:
    //...
    
 variables:
 long time(time);
    //...
 char date_time_string(time,ndate);
    //...
 float latitude(time);
    //... 
 float longitude(time);
    //...

 float conductivity(time);
    conductivity: long_name="conductivity of seawater";
    conductivity: standard_name="conductivity";
    conductivity: units="siemans per meter";

 char conductivity_qcflag(time);
    conductivity_qcflag: long_name="specific test quality control flag for conductivity";
    conductivity_qcflag: standard_name="conductivity_qcflag";
    conductivity_qcflag: c_format = "%s:%s:%s:%s";
    conductivity_qcflag: aggregate_quality_level: "3";
    conductivity_qcflag: qc_documentation_url: "http://www.qartod/handbook.pdf, http://www.mysite/myimplementation.html"; 

    //if the below variable attributes are not defined here, they should be in the qc_documentation_url

    conductivity_qcflag: sensor_model="platinum-electrode glass cell";
    conductivity_qcflag: sensor_serial_num="21003";
    conductivity_qcflag: accuracy=".001";
    conductivity_qcflag: resolution".0001";
    conductivity_qcflag: valid_instrument_range=."0.,7.";
    conductivity_qcflag: max_continuous_change="4.";
    conductivity_qcflag: geographic_range="0.,7.";
    conductivity_qcflag: dependency = "measurement";
    conductivity_qcflag: reference="none";
    conductivity_qcflag: sampling_interval="10";
    conductivity_qcflag: averaging_interval="0";
    conductivity_qcflag: physical_calibration="yes";
    conductivity_qcflag: last_cal_date="21-Oct-2000";

 // global attributes
 :conventions = "CF-1.0, SEACOOS-NETCDF-2.0, SEACOOS-QC-1.0";

 // global variable attribute defaults - variable attribute defaulted to the below if not defined within the variable

 // aggregate_quality_level values
 // -9 = missing value
 // 0 = quality not evaluated
 // 1 = bad
 // 2 = questionable/suspect
 // 3 = good

 :aggregate_quality_level: "3";
 :qc_documentation_url: "http://www.qartod/handbook.pdf, http://www.mysite/myimplementation.html"; 


data:

time = 1052557202,1052557214,1052557226, ...

date_time_string = "2003 05 10 09 00 02","2003 05 10 09 00 14","2003 05 10 09 00 26", ...

latitude = 25.780594,25.780422,25.780250, ...

longitude = -80.038314,-80.038876,-80.039430, ...

conductivity = 5.7817,5.7825,5.7831, ...

conductivity_qcflag = "O:R:C:G", "O:r:c:g", "o:x:x:x", "O:R:C:g 1", "O:R:C:G", ...

}

Sample qc netcdf description

Convention listing

The netcdf convention for the seacoos qc is listed with the other conventions in the global conventions attribute (see also Unidata documentation regarding netcdf conventions at http://my.unidata.ucar.edu/content/software/netcdf/conventions.html).

 // global attributes
 :conventions = "CF-1.0, SEACOOS-NETCDF-2.0, SEACOOS-QC-1.0"; 

global variable attribute defaults

The below concept is something I was thinking of as a shortcut from having to list a common attribute several times for each variable. It's in between a global attribute which applies to the whole file and a variable attribute which applies to a single variable. Basically if the attribute is listed at the variable level, it takes precedence over the global listing, but if not listed, the global attribute is defaulted.

Creating global variable attribute defaults for the aggregate_quality_level and qc_documentation_url.

Spelling out the aggregate quality level listing same as the Qartod listing. At this point unsure of how these values are applied apart from the individual data points - for instance, does questionable/suspect or bad apply to all the values if only a single value is bad? Or is there a certain number/threshold of observations which are 'off' before the aggregate level is labeled a certain way?

The qc_documentation_url would contain a list of url's where lookup tables, algorithms and other additional data which we haven't come up with a quick way of representing within the netcdf would be listed. The content of these urls might be redundant, providing a central repository listing say at qartod.org as well another copy available at the data provider site.

 // global variable attribute defaults - variable attribute defaulted to the below if not defined within the variable

 // aggregate_quality_level values
 // -9 = missing value
 // 0 = quality not evaluated
 // 1 = bad
 // 2 = questionable/suspect
 // 3 = good

 :aggregate_quality_level: "3";
 :qc_documentation_url: "http://www.qartod/handbook.pdf, http://www.mysite/myimplementation.html"; 

Variable qc attributes

After the usual listing of a variable name and attributes, the corresponding qc variable and attributes are listed. The qc variable name and standard name follows the same name as the variable with the suffix of '_qcflag'. Of the first 5 qc variable attributes listed in the below, the first 2(long_name, standard_name) would always be present and the other 3 would(c_format,aggregate_quality_level,qc_documentation_url) always be listed at either the qc variable attribute or defaulted from the global variable attribute.

I think the other qc variable attributes starting with 'sensor_model' would be good to have, but their presence would not be as critical as the basic qc_flags and might possibly be offset to the qc_documentation_url depending on the complexity of the description. I've added 'max_continuous_change' to the qc attribute list Liz has started and am looking for Liz to provide some more definition where I'm not sure what the attribute means.

  • sensor_model="platinum-electrode glass cell";
Sensor model description

  • sensor_serial_num="21003";
Sensor serial number

  • accuracy=".001";
Decimal amount within which this observation should be accurate.

  • resolution".0001";
Decimal capable resolution of the observation.

  • valid_instrument_range=."0.,7.";
Range of the observation capable by the sensor(independent of environment).

  • max_continuous_change="4.";
Overall/annual maximum change(+/-) from one observation point to the next.

  • geographic_range="0.,7.";
Overall/annual range of the observation(dependent on the environment).

  • dependency = "measurement";
??

  • reference="none";
??

  • sampling_interval="10";
Number of minutes between which observations are collected.

  • averaging_interval="0";
Number of minutes over which the observed data has been averaged.

  • physical_calibration="yes";
??

  • last_cal_date="21-Oct-2000";
Date of the last calibration of the sensor

 float conductivity(time);
    conductivity: long_name="conductivity of seawater";
    conductivity: standard_name="conductivity";
    conductivity: units="siemans per meter";

 char conductivity_qcflag(time);
    conductivity_qcflag: long_name="specific test quality control flag for conductivity";
    conductivity_qcflag: standard_name="conductivity_qcflag";
    conductivity_qcflag: c_format = "%s:%s:%s:%s";
    conductivity_qcflag: aggregate_quality_level: "3";
    conductivity_qcflag: qc_documentation_url: "http://www.qartod/handbook.pdf, http://www.mysite/myimplementation.html"; 

    //if the below variable attributes are not defined here, they should be in the qc_documentation_url

    conductivity_qcflag: sensor_model="platinum-electrode glass cell";
    conductivity_qcflag: sensor_serial_num="21003";
    conductivity_qcflag: accuracy=".001";
    conductivity_qcflag: resolution".0001";
    conductivity_qcflag: valid_instrument_range=."0.,7.";
    conductivity_qcflag: max_continuous_change="4.";
    conductivity_qcflag: geographic_range="0.,7.";
    conductivity_qcflag: dependency = "measurement";
    conductivity_qcflag: reference="none";
    conductivity_qcflag: sampling_interval="10";
    conductivity_qcflag: averaging_interval="0";
    conductivity_qcflag: physical_calibration="yes";
    conductivity_qcflag: last_cal_date="21-Oct-2000";

Explanation of flagging scheme

The syntax structure of the flags is captured in the c_format qc attribute similar to

conductivity_qcflag: c_format = "%s:%s:%s:%s";

which implies that there will be 4 sets of flags, each flag string separated by a colon. A string may be a single character or several characters using the space character as a separator.

The flag scheme proposed uses uppercase characters to signify passing values and lowercase characters to signify failing values. All proposed flags need not necessarily be evaluated.

O=sensor working properly
o=sensor malfunction or readings needs further evaluation

Note that for this initial flag, if a sensor is malfunctioning, then 'x' can be substituted for subsequent dependent flags to represent that the test was not evaluated. For example a reading might look like "o:x:x".

Note for the below range(R) and continuity(C) flags that the subconditions are the same except that continuity has an extra condition for flatlining.

R=observation is within geographic/environment range

  • subflags for passing:
    • 1=observation would normally fail, but passes as part of unusual event (hurricane, etc)

r=observation is outside of geographic/environment range

  • subflags for failure:
    • 1=observation is outside of seasonal range
    • 2=observation is outside of annual range
    • 3=observation is outside of sensor range
    • 4=observation is outside of internal consistency range
    • 5=observation is outside of geographic consistency range

C=observation is within continuity range

  • subflags for passing:
    • 1=observation would normally fail, but passes as part of unusual event (hurricane, etc)

c=observation is outside of continuity range

  • subflags for failure:
    • 1=observation is outside of seasonal range
    • 2=observation is outside of annual range
    • 3=observation is outside of sensor range
    • 4=observation is outside of internal consistency range
    • 5=observation is outside of geographic consistency range
    • 6=observation has flatlined or shows less variation than normal

Suggested additional flags (if applicable). Documentation for tests included in the attribute qc_documentation_url.

For the following netcdf example flag readings:
The first reading "O:R:C" indicates the sensor online, passing the range and continuity interval tests.

The second reading "O:r:c" indicates the sensor online, but failing the range and continuity interval tests.

The third reading "o:x:x" indicates the sensor offline and all subsequent flags not evaluated.

The fourth reading "O:r 1:C" indicates the sensor online, passing the continuity interval test, but failing the range test with a value outside of the seasonal limit.

conductivity = 5.7817,5.7825,5.7831, ...

conductivity_qcflag = "O:R:C", "O:r:c", "o:x:x", "O:r 1:C", "O:R:C", ...

Update: December 5, 2005

Following up a message by Payne saying that I see the following 3 general options for how we format the low level flags:

For the low level flags, 1=pass test, 0=fail test

#Most descriptive and least compacted
1) Define separate 'shadow' qc variables for each observation_type(sst for example) with one 'shadow' variable for each type of low level test (currently 4 qc variables to every variable populated 1 or 0 for each data point.).

salinity = 36.3266,36.3288,36.3292, ...
salinity_qcflag_high_level = 3, 3, 0, ...
salinity_qcflag_equipment_range = 1, 1, 0, ...
salinity_qcflag_local_range = 1, 1, 0, ...
salinity_qcflag_lack_of_data = 1, 1, 0, ...
salinity_qcflag_time_discrepancy = 1, 1, 0, ...

#Still descriptive, but more compacted.
2a) Define a single 'shadow' qc variable for each observation_type which represents the 4 low level flags as a sub-array of agreed test order(equipment_range,local_range,missing,temporal_continuity).

salinity = 36.3266,36.3288,36.3292, ...
salinity_qcflag_high_level = 3, 3, 0, ...
salinity_qcflag[0,1,2,3] = "1","1","1","0" , "0","0","0","0" , "1","1","0","0" ...

2b) Define a single 'shadow' qc variable for each observation_type which represents the string concatenation of agreed test order(equipment_range,local_range,missing,temporal_continuity).

salinity = 36.3266,36.3288,36.3292, ...
salinity_qcflag_high_level = 3, 3, 0, ...
salinity_qcflag = "1:1:1:0", "0:0:0:1", "1:1:0:0", ...

#This option is most compacted and you'd need to convert from decimal to binary to interpret the test results.
3)Define a single 'shadow' qc variable for each observation_type which represents the concatenation of agreed test order(equipment_range,local_range,missing,temporal_continuity) converted from a binary (1001 for example) to int (9 for example).

salinity = 36.3266,36.3288,36.3292, ...
salinity_qcflag_high_level = 3, 3, 0, ...
salinity_qcflag = "9", "3", "7", ...

Payne mentioned preferring option #2b, I'd probably go with something like #1 since I'd tend to favor separation and readability where the amount of data being processed isn't that much.

March 23, 2005

QC Web Service

After a recent group conference call on qc issues, I was thinking of some possible software/service type solutions for the problem. My initial thoughts are along the lines of a service, but code would be provided if folks want to run it locally as well.

The web service URL parameters could include

  • input netCDF file or ascii column oriented flatfile
  • a XML descriptor file which describes the layout of the netCDF file(this could also be created as part of the Seacoos netCDF convention->XML translation proposed earlier), identifiers for internal consistency checks, identifiers for offline instruments
  • a XML variable->limits lookup file associated with the data file for calculating range checks
  • a XML primer file with earlier values if needed for continuity or other checks which need previous values
  • a XML file representing other local geographic values for variables for geographic consistency checks

Submitting these paremeters would use the input netCDF file along with the other supplied XML files to create an output netCDF or ascii column oriented flatfile with the appropriate qcflags included within the file. Data providers have dynamic control of the processing using various input files which could be easily adopted by other users.

For XML reading and writing, I'd recommend use of the perl XML::Simple package http://homepages.paradise.net.nz/gmclean1/cpan/

and continuing use of the perl netCDF and udunits packages/libraries.

Update: As of recently I'd recommend looking at XML::XPath instead of XML::Simple as it uses cleaner, more standardized notation for navigating xml documents. XML::XPath is docomented at http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/JCNotes#September_22_2005_Perl_XML_XPath

QC Web Service - Simple Example

I've modified the DODSFilterColumn service (code here)to apply a basic range check on temperature for an annual high and low. Note that the below version is for demo purposes only, it assumes that temp is the only column passed to it. The service will assign a passing range flag of 'R' or a failing range annual flag of 'r 2' using the qc document listed below. The service will accept any xml source with the same document structure if folks want to substitute their own ranges and data sources to see the effect.

The http arguments passed to the service are

  • dods_url - a DODS netCDF server URL with a single variable request representing temperature
  • time_units - the 'time' unit declaration for the netCDF
  • qc_source - a xml source with the element layout same as the below document

Potentially an ASCII column data source could also be passed as a data source using an xml descriptive file to explain the ASCII column layout similar to the effort with SeacoosXMLServices

Sample URL
http://nautilus.baruch.sc.edu/services/seacoos_qc.php?dods_url=http://seacoos.skio.peachnet.edu/cgi-bin/nph-dods/proc_data/r2/near_bot/R2PkgB_1999_09.nc.ascii?temp&time_units=days%20since%20-0001-1-2%2000:00:00&qc_source=http://nautilus.baruch.sc.edu/services/qc.xml

qc document (xml)

http://nautilus.baruch.sc.edu/services/qc.xml

<?xml version="1.0" encoding="UTF-8"?>
<xml>

<global_attributes>

   <convention>seacoos_qc_xml_1.0</convention>

</global_attributes>

<observation_types>

   <sea_surface_temperature>
      <range_test>
         <unit>degree_Celsius</unit>
      
         <seasonal>
            <start_date>01-01</start_date>
               <high>21.1</high>
               <low>15.5</low>
            <stop_date>03-31</stop_date>
            
            <start_date>04-01</start_date>
               <high>23.8</high>
               <low>15.5</low>
            <stop_date>06-30</stop_date>
            
            <start_date>07-01</start_date>
               <high>29.4</high>
               <low>18.3</low>
            <stop_date>09-30</stop_date>            

            <start_date>10-01</start_date>
               <high>29.4</high>
               <low>18.3</low>
            <stop_date>12-31</stop_date>
         </seasonal>

         <annual>
            <high>29.4</high>
            <low>15.5</low>
         </annual>
         
         <sensor>
            <high>40.0</high>
            <low>0.0</low>
         </sensor>
      </range_test>
      
      <continuity_test>
         <unit>degree_Celsius hour-1</unit>
         
         <seasonal>
            <start_date>01-01</start_date>
               <maximum_change>1.0</maximum_change>
            <stop_date>03-31</stop_date>      
      
            <!-- ... -->
      
         </seasonal>

         <!-- ... -->
         
      </continuity_test>
   </sea_surface_temperature>

   <salinity>
   
      <!-- ... -->
      
   </salinity>

</observation_types>
 
</xml>

to top

I Attachment Action Size Date Who Comment
seacoos_qc.pl.txt manage 6.7 K 24 Jul 2005 - 18:20 JeremyCothran NA

You are here: Main > QCNotes

to top

Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding DMCC? Send feedback