Time of Observation Bias in temperature records: an introductory review

In a two day sample which is displayed further down in this page, a sequence of temperatures occurs which can lead to TOB. This particular sample includes both low morning temperatures, and high evening temperatures, of one day which can affect 24 hour min/max readings of the next day. It would be fairly common to see one, or the other, rather than both, in any given pair of days. In this instance, day two had a higher low, and a lower high, than day one.

On day 1, the temperature drops gradually, reaching 22 at 8 AM, then climbs to 31 by 11 AM, and then gradually drops such that it was 30 at 4 and 5 PM, and 29 at 6 and 7 PM.

On day 2, the morning low was 26, but 24 hour min/max readings at 6, 7, and 8 AM would all have indicated a low of 22 from the previous morning. The high during day 2 was 28, but that was topped by the higher highs from day 1 for the 4, 5, 6, and 7 PM 24 hour min/max readings.

At any particular weather observation station using a min/max thermometer set, the readings would occur at one particular time each day. Depending on what that time of day is, that station's readings would usually be affected more by either previous day lows, or previous day highs.

Since TOB is here calculated relative to midnight, monthly averages of morning TOB will often be negative, while monthly averages of after noon TOB seem always to be positive (at least among the stations in this review). Also, because TOB is calculated relative to midnight, the largest magnitudes of cool morning TOB will usually be smaller than the largest magnitudes of warm after noon TOB.
For each station in this study, there are two files: one containing hourly temperatures and 24 hour estimates, and the other containing monthly, and annual, estimates/summaries. I'll refer to the former as DAT files, and the latter as SUM files. While the SUM files may be of most interest, the DAT files are provided for those interested in the details on which summaries are based. Except for one sample, the DAT files are compressed as ZIP files due to their relatively large sizes. A link to a list of links to all of those files may be found near the bottom of this page.
Format of the DAT files:

One line for each hourly observation. On all lines: year, month, day, hour, temperature (in degrees Fahrenheit). At midnight, and at several hypothetical times of observation, there will be seven additional numbers:

min (minimum,low) temperature of the past 24 hours
max (maximum,high) temperature of the past 24 hours
(min+max)/2 (mean) of the past 24 hours
average (smoothed) hourly temperature of the past 24 hours
number of consecutive hourly observations in the past 24 hours
hour of most recent occurrence of min temperature in the past 24 hours
hour of most recent occurrence of max temperature in the past 24 hours

Those descriptions need further qualification. "The past 24 hours" will include the observations at both the beginning and the end of those 24 hours, and so will include 25 observations unless some data are missing. The "average (smoothed) hourly temperature of the past 24 hours" uses half of the first, and half of the last, of those observations (plus all of the other 23 observations). The number of consecutive observations will usually be 25, and if it is not, some data are missing, and 24 hour periods that are missing data will not be used in the summaries. If the "hour of most recent occurrence" is 25, it indicates that that occurrence was the observation at the beginning of that 24 hour period (i.e. it was 24 hours old).

For programming convenience, midnight is hour 24 rather than hour 0.

In addition to the numbers, some lines will also have low/high comments (LH comments) which call attention to some particular aspect about the information in that line. The comments, and their meanings are:

L24 : the current 24 hour min (Low) temperature is 24 hours old
H24 : the current 24 hour max (High) temperature is 24 hours old
LPAM : the current 24 hour min (Low) is from the Previous day's AM
HPPM : the current 24 hour max (High) is from the Previous day's PM
The above four comments are most pertinent to TOB. The following eight may be less so, but perhaps of some interest.
L24MN : same as L24 but at midnight
H24MN : same as H24 but at midnight
MidnightLow : this midnight temperature is the most recent occurrence of this day's min
MidnightHigh : this midnight temperature is the most recent occurrence of this day's max
LPPM : the current 24 hour min (Low) is from the Previous day's PM
HTAM : the current 24 hour max (High) is from the This day's AM
NoonHigh : the noon temperature was the most recent occurrence of this day's max
NoonLow : the noon temperature was the most recent occurrence of this day's min

In those comments, noon is treated as AM, and midnight as PM.

The L24, H24, L24MN, and H24MN, are implicitly counted in the summaries by time of observation. A different kind of summary of all of the LH comments is also in the SUM file, and is described below.
A two day sample (discussed at the top of this page) from a DAT file:
(Unless otherwise indicated, the samples are from IL14842, Peoria, Illinois)

84 01 07 24 029  23  37  30.0  30.333 25  3 16
84 01 08  1 029
84 01 08  2 029
84 01 08  3 027
84 01 08  4 027
84 01 08  5 026
84 01 08  6 024  24  37  30.5  30.833 25  6 16
84 01 08  7 023  23  37  30.0  30.667 25  7 16
84 01 08  8 022  22  37  29.5  30.521 25  8 16
84 01 08  9 027  22  37  29.5  30.417 25  8 16
84 01 08 10 029
84 01 08 11 031
84 01 08 12 031
84 01 08 13 031
84 01 08 14 030
84 01 08 15 030
84 01 08 16 030  22  37  29.5  29.063 25  8 25 H24
84 01 08 17 030  22  35  28.5  28.813 25  8 25 H24
84 01 08 18 029  22  33  27.5  28.625 25  8 25 H24
84 01 08 19 029  22  31  26.5  28.521 25  8 13
84 01 08 20 028  22  31  26.5  28.479 25  8 13
84 01 08 21 027  22  31  26.5  28.375 25  8 13
84 01 08 22 027
84 01 08 23 026
84 01 08 24 026  22  31  26.5  27.896 25  8 13
84 01 09  1 026
84 01 09  2 027
84 01 09  3 028
84 01 09  4 028
84 01 09  5 027
84 01 09  6 027  22  31  26.5  27.813 25  8 13 LPAM
84 01 09  7 027  22  31  26.5  27.958 25  8 13 LPAM
84 01 09  8 027  22  31  26.5  28.146 25 25 13 L24
84 01 09  9 027  26  31  28.5  28.250 25  1 13
84 01 09 10 028
84 01 09 11 028
84 01 09 12 028
84 01 09 13 027
84 01 09 14 027
84 01 09 15 026
84 01 09 16 026  26  30  28.0  27.417 25 16 17 HPPM
84 01 09 17 027  26  30  28.0  27.271 25 16 25 H24
84 01 09 18 027  26  29  27.5  27.167 25 16 19 HPPM
84 01 09 19 026  26  29  27.5  27.063 25 19 25 H24
84 01 09 20 025  25  28  26.5  26.938 25 20 12 HTAM
84 01 09 21 024  24  28  26.0  26.813 25 21 12 HTAM
84 01 09 22 024
84 01 09 23 024
84 01 09 24 023  23  28  25.5  26.479 25 24 12 MidnightLow NoonHigh

The DAT files become the input for the corresponding SUM files. The SUM files have a variety of summaries.

There are summaries of individual months, each consisting of six lines. Each line contains:

year, month, flag, followed by monthly averages, or counts, of quantities indicated by the flag for midnight and several hypothetical times of observation.

The flags are:
T : average of daily (min+max)/2 for 24 hour periods ending at the indicated hour during that month (including midnight of the last day of that month)
B : estimated average TOB relative to midnight of (min+max)/2 for 24 hour periods ending at the indicated hour (except for the midnight column which is simply carried down from the T line). The monthly average of daily midnight (min+max)/2 is subtracted from the monthly average (min+max)/2 of each hypothetical time of observation.
A : average of the daily smoothed average hourly readings for 24 hour periods ending at the indicated hour
D : number of 25 consecutive observation sets for 24 hour periods ending at the indicated hour (if these numbers are less than the number of days in that month, then some data are missing)
L : number of times the min (Low) was 24 hours old as of the indicated hour
H : number of times the max (High) was 24 hours old as of the indicated hour
A sample monthly summary:

YR  M F   24:00   06:00   07:00   08:00   09:00   16:00   17:00   18:00   19:00
 
86 05 T  63.806  62.984  63.597  63.871  64.016  65.032  64.871  64.661  64.419
86 05 B  63.806  -0.822  -0.209   0.065   0.210   1.226   1.065   0.855   0.613
86 05 A  63.935  63.676  63.681  63.694  63.709  63.820  63.841  63.862  63.884
86 05 D      31      31      31      31      31      31      31      31      31
86 05 L       0      12       7       4       0       0       0       0       0
86 05 H       1       0       0       0       0       7       7       5       3
This summary indicates, for example, that at that station in May 1986, 24 hour (min+max)/2 observations at 06:00 would indicate a monthly mean temperature 0.8° F lower than 24 hour (min+max)/2 observations at midnight, while such observations at 16:00 would indicate a monthly mean temperature 1.2° F higher than at midnight.
Based on the summaries of individual months, but preceding them in the SUM files, are summaries of estimated TOB, and then two summaries of 24 hour old min/max counts. There is also a summary of LH comments.

The first TOB summary consists of annual averages of monthly TOB.

The second TOB summary consists of month of year averages for each of the 12 calendar months (i.e. averages of the "B" lines of all of the Januarys, then of all of the Februarys, etc.).

The third TOB summary is a single line summary summary. (This line is gathered from all of the station SUM files and placed together in two files mentioned near the bottom of this page.)

In each of these summaries the numbers under the 24:00 heading are the estimated average temperatures of the respective periods.

The 24 hour old min/max (Low;High) summaries are simply totals of the L and H lines of the individual month summaries. The first totals by month of the year, the second totals by year. In these summaries, if a letter appears where you expected a number, the letter A=10, B=11, C=12, etc.

While 24 hour old lows and highs contribute to TOB, it will be clear that the numbers of their occurences are not proportional to the magnitudes of TOB.
Sample TOB summaries:

   YR  24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
AB 84  51.09 -0.63 -0.37 -0.09  0.15  1.45  1.21  0.98  0.78  0.61  0.48
AB 85  49.70 -0.67 -0.42 -0.09  0.24  1.53  1.29  1.03  0.77  0.59  0.44
AB 86  51.81 -0.61 -0.36 -0.04  0.29  1.63  1.33  1.00  0.75  0.58  0.42
AB 87  53.64 -0.62 -0.38 -0.03  0.23  1.53  1.25  0.93  0.75  0.59  0.43
AB 88  51.64 -0.70 -0.43 -0.14  0.20  1.74  1.42  1.11  0.85  0.64  0.47
AB 89  49.29 -0.68 -0.41 -0.11  0.27  1.71  1.41  1.15  0.90  0.71  0.54
AB 90  52.12 -0.47 -0.20  0.11  0.39  1.75  1.48  1.13  0.84  0.62  0.45
AB 91  52.62 -0.62 -0.38 -0.06  0.24  1.63  1.26  0.92  0.63  0.47  0.34
AB 92  51.28 -0.54 -0.24  0.06  0.31  1.54  1.24  0.94  0.70  0.54  0.37
Small year to year variations of annual TOB averages are common.
   MN  24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
AB 01  25.80 -0.94 -1.01 -0.95 -0.44  1.75  1.43  1.14  0.93  0.69  0.53
AB 02  29.09 -1.00 -1.10 -0.75 -0.12  1.71  1.39  1.08  0.87  0.68  0.50
AB 03  40.99 -1.22 -1.05 -0.45  0.01  1.97  1.67  1.27  0.97  0.75  0.54
AB 04  52.59 -1.16 -0.50 -0.05  0.14  1.87  1.67  1.32  0.94  0.67  0.49
AB 05  62.92 -0.68 -0.03  0.17  0.32  1.58  1.35  1.09  0.79  0.60  0.42
AB 06  71.82 -0.20  0.14  0.31  0.41  1.32  1.16  0.90  0.63  0.45  0.36
AB 07  75.07 -0.21  0.14  0.24  0.30  1.05  0.84  0.68  0.48  0.33  0.25
AB 08  72.51 -0.29  0.26  0.39  0.42  1.18  1.01  0.72  0.50  0.39  0.30
AB 09  65.57 -0.36  0.18  0.60  0.71  1.60  1.30  0.89  0.66  0.55  0.41
AB 10  53.09 -0.69 -0.48  0.23  0.57  1.75  1.32  0.98  0.80  0.63  0.49
AB 11  40.22 -0.39 -0.47 -0.10  0.40  1.85  1.49  1.23  1.01  0.83  0.58
AB 12  27.93 -0.26 -0.31 -0.17  0.36  1.72  1.26  0.96  0.74  0.56  0.37
Large seasonal, or month of year, differences of TOB averages are common.
 Years 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
 9 Yrs 51.47 -0.62 -0.35 -0.04  0.26  1.61  1.32  1.02  0.78  0.59  0.44

Sample 24 hour old Low;High summaries:
L;H MN 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
L;H 01 17;20 34;16 60;10 109;B  85;6  1;64  0;55  5;38  4;34  5;31  5;25
L;H 02  7;17 33;10  87;9  94;6  65;4  2;76  2;64  4;46  3;36  5;31  5;30
L;H 03  6;19  69;8 110;7  89;8 51;11  0;73  1;75  0;60  0;42  1;36  4;23
L;H 04   7;8 107;1  77;3  49;4  23;4  0;58  0;74  0;61  0;45  0;27  2;22
L;H 05   2;5 111;2  48;1  22;2   9;1  0;60  1;67  0;55  0;33  1;17  1;10
L;H 06   1;1  68;0  33;0  17;1   5;2  1;57  0;71  0;58  0;30  0;11   1;9
L;H 07   4;1  96;0  27;0  11;0   7;0  0;68  0;51  0;45  0;29  0;12   0;5
L;H 08   1;2 108;0  41;0  10;0   5;1  1;66  1;67  1;41  0;25  0;10   0;6
L;H 09   2;6 111;1  82;0  31;3  13;4  1;81  2;72  1;42  0;17  0;14  3;13
L;H 10  3;10  84;3 110;4  63;4  22;1  1;82  0;62  0;34  2;28  8;23  2;18
L;H 11  8;22 40;16 103;D 92;10  71;6  2;69  3;49  4;38  7;31  5;28  4;27
L;H 12 13;26 26;11 69;12 108;B 81;11  4;74  5;49  8;40 15;32 10;25 18;24
As with TOB, large seasonal, or month of year, differences of L;H counts are common.
L;H YR 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
L;H 84 10;19  99;7  89;6 77;10  55;8  3;82  1;77  2;58  2;45  6;37  5;32
L;H 85  7;14 101;B  95;6  78;7 58;10  3;92  4;85  4;66  5;38  9;28  9;20
L;H 86  7;13  96;9  96;7  83;6  54;3  0;96  1;93  1;57  0;41  0;25  3;15
L;H 87  7;15  94;8  95;5  74;6  36;6  1;87  3;81  3;52  3;37  3;27  6;24
L;H 88 12;12  96;5  90;9  71;6  46;8  0;96  2;88  2;66  6;45  2;31  3;26
L;H 89  8;15  94;8 100;6  80;5  53;4  0;94  1;78  3;61  3;49  3;35  2;28
L;H 90  8;18  94;8 100;A 79;10  42;5  2;83  0;83  0;68  2;50  2;36  7;20
L;H 91  7;14 106;6  93;6  75;5  44;4 2;108  1;84  4;71  6;39  6;19  5;24
L;H 92  5;17 107;6  89;4  78;5  49;3  2;90  2;87  4;59  4;38  4;27  5;23
Again as with TOB, year to year variations of annual L;H counts are common.
The summary of LH comments consists of counts of the comments in the DAT file, counts of the number of days on which each comment occurred for those comments which could occur more than once per day, plus a few other counts. With shorter abbreviations for a few of the comments, their meanings are:

H24 : the current 24 hour max (High) temperature is 24 hours old
HPPM : the current 24 hour max (High) is from the Previous day's PM
L24 : the current 24 hour min (Low) temperature is 24 hours old
LPAM : the current 24 hour min (Low) is from the Previous day's AM
HTAM : the current 24 hour max (High) is from the This day's AM
LPPM : the current 24 hour min (Low) is from the Previous day's PM
H24MN : same as H24 but at midnight
L24MN : same as L24 but at midnight
MdnHi : this midnight temperature is the most recent occurrence of this day's max
MdnLo : this midnight temperature is the most recent occurrence of this day's min
NnH : the noon temperature was the most recent occurrence of this day's max
NnL : the noon temperature was the most recent occurrence of this day's min

LHdays is the number of days in which any LH comment occurred
MLHdays is the number of days in which LH comments occurred in multiple hours
LHhours is the number of hours for which any LH comment occurred
Sample LH comments summary:
       H24  HPPM   L24  LPAM  HTAM  LPPM H24MN L24MN MdnHi MdnLo  NnH  NnL
Cts   3001  1085  2866   953  1073   328   137    71    99   834   94    4
Days  1106   514  1434   648   239   118
 
Total Days= 3287 LHdays= 2383 MLHdays= 1956
Total Hours= 78887 LHhours= 10401

Some samples of TOB summaries from selected stations:
Station Years 24:00 06:00 07:00 08:00 09:00 16:00 17:00 18:00 19:00 20:00 21:00
 
AL03856 8 Yrs 61.08 -0.56 -0.30  0.07  0.31  1.26  0.98  0.74  0.57  0.45  0.35
CA23174 9 Yrs 63.22 -0.24 -0.10  0.06  0.20  0.21  0.13  0.10  0.08  0.07  0.05
DE13781 8 Yrs 54.73 -0.59 -0.32  0.01  0.33  1.47  1.20  0.91  0.71  0.57  0.43
FL12836 9 Yrs 78.06 -0.19 -0.12  0.03  0.14  0.39  0.29  0.20  0.15  0.12  0.09
GA03813 9 Yrs 65.04 -0.76 -0.65 -0.30  0.05  1.29  1.08  0.78  0.56  0.42  0.32
IA14933 9 Yrs 50.61 -0.79 -0.59 -0.31  0.03  1.68  1.40  1.08  0.81  0.61  0.46
KS13984 9 Yrs 54.31 -0.84 -0.69 -0.38 -0.00  1.82  1.56  1.16  0.80  0.56  0.40
LA03937 9 Yrs 68.17 -0.51 -0.33 -0.01  0.23  0.95  0.75  0.55  0.43  0.34  0.27
MA14739 9 Yrs 51.22 -0.50 -0.31 -0.03  0.31  1.37  1.14  0.88  0.68  0.50  0.37
NC03812 8 Yrs 55.37 -0.72 -0.60 -0.31  0.07  1.41  1.12  0.80  0.59  0.45  0.34
NE24023 9 Yrs 49.25 -0.77 -0.56 -0.21  0.15  2.08  1.79  1.33  0.89  0.63  0.45
OK13967 9 Yrs 60.47 -0.71 -0.60 -0.33  0.01  1.50  1.28  0.93  0.64  0.46  0.34
PA04751 9 Yrs 44.83 -0.68 -0.39 -0.06  0.24  1.69  1.47  1.20  0.93  0.70  0.50
RI14765 8 Yrs 51.30 -0.41 -0.15  0.16  0.48  1.39  1.12  0.86  0.66  0.52  0.38
SD14936 9 Yrs 46.15 -0.75 -0.51 -0.16  0.16  2.00  1.72  1.33  0.96  0.70  0.50
TN13877 9 Yrs 56.13 -0.78 -0.64 -0.33  0.03  1.45  1.23  0.92  0.69  0.52  0.40
UT24127 9 Yrs 52.52 -0.44 -0.17  0.08  0.25  1.15  0.95  0.70  0.51  0.37  0.28
VA13737 9 Yrs 60.53 -0.45 -0.21  0.12  0.44  1.48  1.22  0.92  0.69  0.51  0.37
WI14837 9 Yrs 46.57 -0.54 -0.20  0.11  0.41  1.64  1.38  1.11  0.86  0.65  0.50

Source of hourly data: http://www.epa.gov/scram001/surfacemetdata.htm of which data from 190 stations were used, including 9 years of data for 171 stations, and 8 years of data for 19 stations.

Stations used list : LOCATION.STA
190 TOB summaries : AVGTOBF.SMY
Those files combined: AVGTOBF.STA
SUM & DAT file links: SUMDATF.HTM

Comments are welcome, especially if the above descriptions of the content of the files are unclear, if there are errors in any of the files, or if there is interest in further details.

Jerry Brennan
November 2005