Time of Observation Bias in temperature records: an introductory review

In a two day sample which is displayed further down in this page, a sequence of temperatures occurs which can lead to TOB. This particular sample includes both low morning temperatures, and high evening temperatures, of one day which can affect 24 hour min/max readings of the next day. It would be fairly common to see one, or the other, rather than both, in any given pair of days. In this instance, day two had a higher low, and a lower high, than day one.

On day 1, the temperature drops gradually, reaching -5.6 at 8 AM, then climbs to -0.6 by 11 AM, and then gradually drops such that it was -1.1 at 4 and 5 PM, and -1.7 at 6 and 7 PM.

On day 2, the morning low was -3.3, but 24 hour min/max readings at 6, 7, and 8 AM would all have indicated a low of -5.6 from the previous morning. The high during day 2 was -2.2, but that was topped by the higher highs from day 1 for the 4, 5, 6, and 7 PM 24 hour min/max readings.

At any particular weather observation station using a min/max thermometer set, the readings would occur at one particular time each day. Depending on what that time of day is, that station's readings would usually be affected more by either previous day lows, or previous day highs.

Since TOB is here calculated relative to midnight, monthly averages of morning TOB will often be negative, while monthly averages of after noon TOB seem always to be positive (at least among the stations in this review). Also, because TOB is calculated relative to midnight, the largest magnitudes of cool morning TOB will usually be smaller than the largest magnitudes of warm after noon TOB.
For each station in this study, there are two files: one containing hourly temperatures and 24 hour estimates, and the other containing monthly, and annual, estimates/summaries. I'll refer to the former as DAT files, and the latter as SUM files. While the SUM files may be of most interest, the DAT files are provided for those interested in the details on which summaries are based. Except for one sample, the DAT files are compressed as ZIP files due to their relatively large sizes. A link to a list of links to all of those files may be found near the bottom of this page.
Format of the DAT files:

One line for each hourly observation. On all lines: year, month, day, hour, temperature (in degrees Celsius, converted from degrees Fahrenheit and rounded to tenths). At midnight, and at several hypothetical times of observation, there will be seven additional numbers:

min (minimum,low) temperature of the past 24 hours
max (maximum,high) temperature of the past 24 hours
(min+max)/2 (mean) of the past 24 hours
average (smoothed) hourly temperature of the past 24 hours
number of consecutive hourly observations in the past 24 hours
hour of most recent occurrence of min temperature in the past 24 hours
hour of most recent occurrence of max temperature in the past 24 hours

Those descriptions need further qualification. "The past 24 hours" will include the observations at both the beginning and the end of those 24 hours, and so will include 25 observations unless some data are missing. The "average (smoothed) hourly temperature of the past 24 hours" uses half of the first, and half of the last, of those observations (plus all of the other 23 observations). The number of consecutive observations will usually be 25, and if it is not, some data are missing, and 24 hour periods that are missing data will not be used in the summaries. If the "hour of most recent occurrence" is 25, it indicates that that occurrence was the observation at the beginning of that 24 hour period (i.e. it was 24 hours old).

For programming convenience, midnight is hour 24 rather than hour 0.

In addition to the numbers, some lines will also have low/high comments (LH comments) which call attention to some particular aspect about the information in that line. The comments, and their meanings are:

L24 : the current 24 hour min (Low) temperature is 24 hours old
H24 : the current 24 hour max (High) temperature is 24 hours old
LPAM : the current 24 hour min (Low) is from the Previous day's AM
HPPM : the current 24 hour max (High) is from the Previous day's PM
The above four comments are most pertinent to TOB. The following eight may be less so, but perhaps of some interest.
L24MN : same as L24 but at midnight
H24MN : same as H24 but at midnight
MidnightLow : this midnight temperature is the most recent occurrence of this day's min
MidnightHigh : this midnight temperature is the most recent occurrence of this day's max
LPPM : the current 24 hour min (Low) is from the Previous day's PM
HTAM : the current 24 hour max (High) is from the This day's AM
NoonHigh : the noon temperature was the most recent occurrence of this day's max
NoonLow : the noon temperature was the most recent occurrence of this day's min

In those comments, noon is treated as AM, and midnight as PM.

The L24, H24, L24MN, and H24MN, are implicitly counted in the summaries by time of observation. A different kind of summary of all of the LH comments is also in the SUM file, and is described below.
A two day sample (discussed at the top of this page) from a DAT file:
(Unless otherwise indicated, the samples are from IL14842, Peoria, Illinois)

84 01 07 24  -1.7  -5.0   2.8  -1.10  -0.925 25  3 16
84 01 08  1  -1.7
84 01 08  2  -1.7
84 01 08  3  -2.8
84 01 08  4  -2.8
84 01 08  5  -3.3
84 01 08  6  -4.4  -4.4   2.8  -0.80  -0.654 25  6 16
84 01 08  7  -5.0  -5.0   2.8  -1.10  -0.746 25  7 16
84 01 08  8  -5.6  -5.6   2.8  -1.40  -0.827 25  8 16
84 01 08  9  -2.8  -5.6   2.8  -1.40  -0.885 25  8 16
84 01 08 10  -1.7
84 01 08 11  -0.6
84 01 08 12  -0.6
84 01 08 13  -0.6
84 01 08 14  -1.1
84 01 08 15  -1.1
84 01 08 16  -1.1  -5.6   2.8  -1.40  -1.644 25  8 25 H24
84 01 08 17  -1.1  -5.6   1.7  -1.95  -1.783 25  8 25 H24
84 01 08 18  -1.7  -5.6   0.6  -2.50  -1.890 25  8 25 H24
84 01 08 19  -1.7  -5.6  -0.6  -3.10  -1.950 25  8 13
84 01 08 20  -2.2  -5.6  -0.6  -3.10  -1.973 25  8 13
84 01 08 21  -2.8  -5.6  -0.6  -3.10  -2.029 25  8 13
84 01 08 22  -2.8
84 01 08 23  -3.3
84 01 08 24  -3.3  -5.6  -0.6  -3.10  -2.292 25  8 13
84 01 09  1  -3.3
84 01 09  2  -2.8
84 01 09  3  -2.2
84 01 09  4  -2.2
84 01 09  5  -2.8
84 01 09  6  -2.8  -5.6  -0.6  -3.10  -2.333 25  8 13 LPAM
84 01 09  7  -2.8  -5.6  -0.6  -3.10  -2.254 25  8 13 LPAM
84 01 09  8  -2.8  -5.6  -0.6  -3.10  -2.150 25 25 13 L24
84 01 09  9  -2.8  -3.3  -0.6  -1.95  -2.092 25  1 13
84 01 09 10  -2.2
84 01 09 11  -2.2
84 01 09 12  -2.2
84 01 09 13  -2.8
84 01 09 14  -2.8
84 01 09 15  -3.3
84 01 09 16  -3.3  -3.3  -1.1  -2.20  -2.546 25 16 17 HPPM
84 01 09 17  -2.8  -3.3  -1.1  -2.20  -2.627 25 16 25 H24
84 01 09 18  -2.8  -3.3  -1.7  -2.50  -2.685 25 16 19 HPPM
84 01 09 19  -3.3  -3.3  -1.7  -2.50  -2.742 25 19 25 H24
84 01 09 20  -3.9  -3.9  -2.2  -3.05  -2.810 25 20 12 HTAM
84 01 09 21  -4.4  -4.4  -2.2  -3.30  -2.879 25 21 12 HTAM
84 01 09 22  -4.4
84 01 09 23  -4.4
84 01 09 24  -5.0  -5.0  -2.2  -3.60  -3.060 25 24 12 MidnightLow NoonHigh

The DAT files become the input for the corresponding SUM files. The SUM files have a variety of summaries.

There are summaries of individual months, each consisting of six lines. Each line contains:

year, month, flag, followed by monthly averages, or counts, of quantities indicated by the flag for midnight and several hypothetical times of observation.

The flags are:
T : average of daily (min+max)/2 for 24 hour periods ending at the indicated hour during that month (including midnight of the last day of that month)
B : estimated average TOB relative to midnight of (min+max)/2 for 24 hour periods ending at the indicated hour (except for the midnight column which is simply carried down from the T line). The monthly average of daily midnight (min+max)/2 is subtracted from the monthly average (min+max)/2 of each hypothetical time of observation.
A : average of the daily smoothed average hourly readings for 24 hour periods ending at the indicated hour
D : number of 25 consecutive observation sets for 24 hour periods ending at the indicated hour (if these numbers are less than the number of days in that month, then some data are missing)
L : number of times the min (Low) was 24 hours old as of the indicated hour
H : number of times the max (High) was 24 hours old as of the indicated hour
A sample monthly summary:

YR  M F   24:00   06:00   07:00   08:00   09:00   16:00   17:00   18:00   19:00
 
86 05 T  17.676  17.216  17.556  17.713  17.794  18.358  18.265  18.147  18.016
86 05 B  17.676  -0.460  -0.120   0.037   0.118   0.682   0.589   0.471   0.340
86 05 A  17.743  17.599  17.601  17.609  17.617  17.679  17.691  17.703  17.714
86 05 D      31      31      31      31      31      31      31      31      31
86 05 L       0      12       7       4       0       0       0       0       0
86 05 H       1       0       0       0       0       7       7       5       3
This summary indicates, for example, that at that station in May 1986, 24 hour (min+max)/2 observations at 06:00 would indicate a monthly mean temperature 0.5° C lower than 24 hour (min+max)/2 observations at midnight, while such observations at 16:00 would indicate a monthly mean temperature 0.7° C higher than at midnight.
Based on the summaries of individual months, but preceding them in the SUM files, are summaries of estimated TOB, and then two summaries of 24 hour old min/max counts. There is also a summary of LH comments.

The first TOB summary consists of annual averages of monthly TOB.

The second TOB summary consists of month of year averages for each of the 12 calendar months (i.e. averages of the "B" lines of all of the Januarys, then of all of the Februarys, etc.).

The third TOB summary is a single line summary summary. (This line is gathered from all of the station SUM files and placed together in two files mentioned near the bottom of this page.)

In each of these summaries the numbers under the 24:00 heading are the estimated average temperatures of the respective periods.

The 24 hour old min/max (Low;High) summaries are simply totals of the L and H lines of the individual month summaries. The first totals by month of the year, the second totals by year. In these summaries, if a letter appears where you expected a number, the letter A=10, B=11, C=12, etc.

While 24 hour old lows and highs contribute to TOB, it will be clear that the numbers of their occurences are not proportional to the magnitudes of TOB.
Sample TOB summaries:

   YR  24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
AB 84  10.61 -0.35 -0.21 -0.05  0.09  0.81  0.67  0.54  0.43  0.34  0.26
AB 85   9.83 -0.37 -0.23 -0.05  0.13  0.85  0.72  0.57  0.43  0.33  0.24
AB 86  11.01 -0.34 -0.20 -0.02  0.16  0.90  0.74  0.55  0.41  0.32  0.23
AB 87  12.02 -0.34 -0.21 -0.02  0.13  0.85  0.70  0.52  0.42  0.33  0.24
AB 88  10.91 -0.39 -0.24 -0.08  0.11  0.97  0.79  0.62  0.47  0.36  0.26
AB 89   9.61 -0.38 -0.23 -0.07  0.15  0.95  0.78  0.64  0.50  0.40  0.30
AB 90  11.17 -0.26 -0.11  0.06  0.21  0.97  0.82  0.63  0.47  0.35  0.25
AB 91  11.45 -0.34 -0.21 -0.03  0.13  0.91  0.70  0.51  0.35  0.26  0.19
AB 92  10.71 -0.30 -0.13  0.03  0.18  0.86  0.69  0.52  0.39  0.30  0.21
Small year to year variations of annual TOB averages are common.
   MN  24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
AB 01  -3.45 -0.52 -0.56 -0.52 -0.24  0.98  0.80  0.63  0.52  0.39  0.29
AB 02  -1.62 -0.56 -0.61 -0.41 -0.07  0.95  0.77  0.60  0.49  0.38  0.28
AB 03   5.00 -0.68 -0.59 -0.25  0.01  1.09  0.93  0.70  0.54  0.41  0.30
AB 04  11.44 -0.65 -0.28 -0.03  0.08  1.04  0.92  0.73  0.52  0.37  0.27
AB 05  17.18 -0.38 -0.02  0.09  0.18  0.88  0.75  0.60  0.44  0.33  0.23
AB 06  22.13 -0.11  0.07  0.17  0.23  0.73  0.64  0.50  0.35  0.25  0.20
AB 07  23.93 -0.12  0.08  0.14  0.17  0.58  0.47  0.38  0.26  0.18  0.14
AB 08  22.51 -0.16  0.14  0.22  0.24  0.66  0.56  0.40  0.28  0.22  0.17
AB 09  18.65 -0.20  0.10  0.33  0.39  0.89  0.72  0.49  0.36  0.30  0.23
AB 10  11.72 -0.38 -0.27  0.13  0.32  0.97  0.74  0.54  0.44  0.35  0.27
AB 11   4.57 -0.21 -0.26 -0.06  0.22  1.03  0.83  0.68  0.56  0.46  0.32
AB 12  -2.26 -0.14 -0.17 -0.10  0.20  0.95  0.70  0.53  0.41  0.31  0.21
Large seasonal, or month of year, differences of TOB averages are common.
 Years 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
 9 Yrs 10.82 -0.34 -0.20 -0.02  0.14  0.90  0.74  0.57  0.43  0.33  0.24

Sample 24 hour old Low;High summaries:
L;H MN 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
L;H 01 17;20 34;16 60;10 109;B  85;6  1;64  0;55  5;38  4;34  5;31  5;25
L;H 02  7;17 33;10  87;9  94;6  65;4  2;76  2;64  4;46  3;36  5;31  5;30
L;H 03  6;19  69;8 110;7  89;8 51;11  0;73  1;75  0;60  0;42  1;36  4;23
L;H 04   7;8 107;1  77;3  49;4  23;4  0;58  0;74  0;61  0;45  0;27  2;22
L;H 05   2;5 111;2  48;1  22;2   9;1  0;60  1;67  0;55  0;33  1;17  1;10
L;H 06   1;1  68;0  33;0  17;1   5;2  1;57  0;71  0;58  0;30  0;11   1;9
L;H 07   4;1  96;0  27;0  11;0   7;0  0;68  0;51  0;45  0;29  0;12   0;5
L;H 08   1;2 108;0  41;0  10;0   5;1  1;66  1;67  1;41  0;25  0;10   0;6
L;H 09   2;6 111;1  82;0  31;3  13;4  1;81  2;72  1;42  0;17  0;14  3;13
L;H 10  3;10  84;3 110;4  63;4  22;1  1;82  0;62  0;34  2;28  8;23  2;18
L;H 11  8;22 40;16 103;D 92;10  71;6  2;69  3;49  4;38  7;31  5;28  4;27
L;H 12 13;26 26;11 69;12 108;B 81;11  4;74  5;49  8;40 15;32 10;25 18;24
As with TOB, large seasonal, or month of year, differences of L;H counts are common.
L;H YR 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
L;H 84 10;19  99;7  89;6 77;10  55;8  3;82  1;77  2;58  2;45  6;37  5;32
L;H 85  7;14 101;B  95;6  78;7 58;10  3;92  4;85  4;66  5;38  9;28  9;20
L;H 86  7;13  96;9  96;7  83;6  54;3  0;96  1;93  1;57  0;41  0;25  3;15
L;H 87  7;15  94;8  95;5  74;6  36;6  1;87  3;81  3;52  3;37  3;27  6;24
L;H 88 12;12  96;5  90;9  71;6  46;8  0;96  2;88  2;66  6;45  2;31  3;26
L;H 89  8;15  94;8 100;6  80;5  53;4  0;94  1;78  3;61  3;49  3;35  2;28
L;H 90  8;18  94;8 100;A 79;10  42;5  2;83  0;83  0;68  2;50  2;36  7;20
L;H 91  7;14 106;6  93;6  75;5  44;4 2;108  1;84  4;71  6;39  6;19  5;24
L;H 92  5;17 107;6  89;4  78;5  49;3  2;90  2;87  4;59  4;38  4;27  5;23
Again as with TOB, year to year variations of annual L;H counts are common.
The summary of LH comments consists of counts of the comments in the DAT file, counts of the number of days on which each comment occurred for those comments which could occur more than once per day, plus a few other counts. With shorter abbreviations for a few of the comments, their meanings are:

H24 : the current 24 hour max (High) temperature is 24 hours old
HPPM : the current 24 hour max (High) is from the Previous day's PM
L24 : the current 24 hour min (Low) temperature is 24 hours old
LPAM : the current 24 hour min (Low) is from the Previous day's AM
HTAM : the current 24 hour max (High) is from the This day's AM
LPPM : the current 24 hour min (Low) is from the Previous day's PM
H24MN : same as H24 but at midnight
L24MN : same as L24 but at midnight
MdnHi : this midnight temperature is the most recent occurrence of this day's max
MdnLo : this midnight temperature is the most recent occurrence of this day's min
NnH : the noon temperature was the most recent occurrence of this day's max
NnL : the noon temperature was the most recent occurrence of this day's min

LHdays is the number of days in which any LH comment occurred
MLHdays is the number of days in which LH comments occurred in multiple hours
LHhours is the number of hours for which any LH comment occurred
Sample LH comments summary:
       H24  HPPM   L24  LPAM  HTAM  LPPM H24MN L24MN MdnHi MdnLo  NnH  NnL
Cts   3001  1085  2866   953  1073   328   137    71    99   834   94    4
Days  1106   514  1434   648   239   118
 
Total Days= 3287 LHdays= 2383 MLHdays= 1956
Total Hours= 78887 LHhours= 10401

Some samples of TOB summaries from selected stations:
Station Years 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
AL03856 8 Yrs 16.16 -0.31 -0.16  0.04  0.17  0.70  0.54  0.41  0.32  0.25  0.19
CA23174 9 Yrs 17.35 -0.13 -0.05  0.03  0.11  0.12  0.07  0.05  0.04  0.04  0.03
DE13781 8 Yrs 12.63 -0.33 -0.18  0.01  0.18  0.82  0.67  0.51  0.40  0.31  0.24
FL12836 9 Yrs 25.59 -0.11 -0.07  0.02  0.08  0.21  0.16  0.11  0.08  0.06  0.05
GA03813 9 Yrs 18.36 -0.42 -0.36 -0.17  0.03  0.71  0.60  0.44  0.31  0.23  0.18
IA14933 9 Yrs 10.34 -0.44 -0.33 -0.17  0.01  0.93  0.78  0.60  0.45  0.34  0.25
KS13984 9 Yrs 12.39 -0.47 -0.38 -0.21 -0.00  1.01  0.87  0.64  0.44  0.31  0.22
LA03937 9 Yrs 20.09 -0.28 -0.18 -0.01  0.13  0.53  0.42  0.31  0.24  0.19  0.15
MA14739 9 Yrs 10.68 -0.28 -0.17 -0.02  0.17  0.76  0.63  0.49  0.38  0.28  0.20
NC03812 8 Yrs 12.98 -0.40 -0.33 -0.17  0.04  0.78  0.62  0.45  0.33  0.25  0.19
NE24023 9 Yrs  9.58 -0.43 -0.31 -0.11  0.08  1.15  0.99  0.74  0.49  0.35  0.25
OK13967 9 Yrs 15.81 -0.39 -0.33 -0.18  0.00  0.83  0.71  0.52  0.35  0.25  0.19
PA04751 9 Yrs  7.13 -0.38 -0.22 -0.03  0.13  0.94  0.81  0.67  0.51  0.39  0.28
RI14765 8 Yrs 10.72 -0.23 -0.08  0.09  0.27  0.77  0.62  0.48  0.37  0.29  0.21
SD14936 9 Yrs  7.86 -0.41 -0.29 -0.09  0.09  1.11  0.96  0.74  0.54  0.39  0.28
TN13877 9 Yrs 13.40 -0.44 -0.36 -0.18  0.02  0.80  0.68  0.51  0.38  0.29  0.22
UT24127 9 Yrs 11.40 -0.25 -0.09  0.05  0.14  0.64  0.53  0.39  0.28  0.20  0.16
VA13737 9 Yrs 15.85 -0.25 -0.12  0.07  0.25  0.82  0.68  0.51  0.38  0.28  0.21
WI14837 9 Yrs  8.10 -0.30 -0.11  0.06  0.23  0.91  0.76  0.62  0.48  0.36  0.28

Source of hourly data: http://www.epa.gov/scram001/surfacemetdata.htm of which data from 190 stations were used, including 9 years of data for 171 stations, and 8 years of data for 19 stations.

Stations used list : LOCATION.STA
190 TOB summaries : AVGTOBC.SMY
Those files combined: AVGTOBC.STA
SUM & DAT file links: SUMDATC.HTM

Comments are welcome, especially if the above descriptions of the content of the files are unclear, if there are errors in any of the files, or if there is interest in further details.

Jerry Brennan
November 2005