Project

General

Profile

Bug #708

SIS to DMC database loses instrument sensitivity

Added by Philip Crotwell over 9 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Start date:
06/09/2015
Due date:
% Done:

0%

Estimated time:
Resolution:

Description

This stationxml comes out of SIS for one of my stations (CO.HAW)

http://files.anss-sis.scsn.org/production/FDSNstationXML/CO/CO_HAW.xml

then goes through the converter (autorun by SIS) to give this dataless:

http://files.anss-sis.scsn.org/production/dataless/CO/CO_HAW.dataless

But going to the DMC fdsn-station ws, after the dataless was loaded into the DMC database, the output stationxml

http://service.iris.edu/fdsnws/station/1/query?net=CO&sta=HAW&loc=00&cha=HH2&starttime=2015-06-07T01:00:00&endtime=2016-06-07T07:00:00&level=response&format=xml&includecomments=true&nodata=404

is wrong in that the <Value> and <Frequency> elements are missing within the <InstrumentSensitivity> element for
channel CO.HAW.00.HH2 (start=2013-10-14).

Oddly the other horizontal channel, HH1, does not seem to have this problem. It, as far as I can tell in PDCC, is exactly the same in terms of both number and contents of the response blockettes. At least from this information, it is very hard to say that this actually is a converter bug or a problem further upstream. StationXML for it is here:

http://service.iris.edu/fdsnws/station/1/query?net=CO&sta=HAW&loc=00&cha=HH1&starttime=2015-06-07T01:00:00&endtime=2016-06-07T07:00:00&level=response&format=xml&includecomments=true&nodata=404

The 4 files from the above URLs are attached. Unfortunately I am not positive that these exact files are the ones loaded into the DMC database. These were generated from SIS on 05-Jun-2015 13:05, but the loaded version in the DMC may have been one generated earlier from SIS, with these regenerated due to bugs found in SIS's stationxml and to some bug fixes in the converter.

CO_HAW.xml (427 KB) CO_HAW.xml SIS output stationXml Philip Crotwell, 06/09/2015 11:57 AM
CO_HAW.dataless (96 KB) CO_HAW.dataless converter output dataless Philip Crotwell, 06/09/2015 11:57 AM
fdsnws-station_2015-06-09T18_57_47.xml (7.6 KB) fdsnws-station_2015-06-09T18_57_47.xml fdsn-station output for just the one problem channel Philip Crotwell, 06/09/2015 11:58 AM
fdsnws-station_2015-06-09T19_10_22.xml (13.3 KB) fdsnws-station_2015-06-09T19_10_22.xml fdsn-station output for the other horizontal without the problem Philip Crotwell, 06/09/2015 12:11 PM
NM.ADSC..HNE_2004.staxml (151 KB) NM.ADSC..HNE_2004.staxml Philip Crotwell, 06/30/2015 08:01 AM
CERI_NM_sm.seed (8.69 MB) CERI_NM_sm.seed Philip Crotwell, 06/30/2015 08:01 AM
NM.C1SC.TR.HNZ.staxml (556 KB) NM.C1SC.TR.HNZ.staxml Philip Crotwell, 07/07/2015 01:43 PM

History

#1 Updated by Philip Crotwell over 9 years ago

Have a look at the stationXML returned from (also attached):
http://service.iris.edu/fdsnws/station/1/query?net=NM&sta=ADSC&loc=--&cha=HNE&starttime=2005-06-01T01:00:00&endtime=2005-06-15T07:00:00&level=response&format=xml&includecomments=true&nodata=404

The XML is invalid with:
cvc-complex-type.2.4.a: Invalid content was found starting with element 'Coefficients'. One of '{WC[##other:"http://www.fdsn.org/xml/station/1"]}' is expected.

Stage 11 starts on line 1514, with <Coefficients> on line 1515 followed by <Numerator> elements down to line 1937 where the <Coefficients> ends, then a <Decimation> and a <StageGain> ending on 1949. But on 1950 is another <Coefficients> element when instead the <Stage> should be ending. This continues down to the bottom of the file where the <Stage> and everything ends.

I believe the original dataless used to load this in the DMC database is here (also attached):
ftp://ftp.ceri.memphis.edu/download/mwithers/dataless/CERI_NM_sm.seed

Looking at this dataless in PDCC, there are 12 stages for this channel. So what is happening, I think, is that part of stage 11 is being overwritten by stage 12?

There may be more to this as I don't see why this channel should have that 12 stage in the original dataless, but regardless it is concerning that the fdsn station web service returned XML that is invalid.

#2 Updated by Philip Crotwell over 9 years ago

One more factoid that may be useful, stage 11 in HNN has 501 Numerators. HNE has 413 in stage 11 and 88 in stage 12. The fact that 413 + 88 = 501 is suspicious.

#3 Updated by Philip Crotwell over 9 years ago

Found one more that could be related, invalid stationxml returned,

http://service.iris.edu/fdsnws/station/1/query?net=NM&sta=C1SC&loc=TR&cha=HNZ&starttime=1997-06-07T01:00:00&endtime=2003-06-07T07:00:00&level=response&format=xml&includecomments=true&nodata=404

Stage 1 has <PolesZeros> then <StageGain> then another <PolesZeros>

Reasonable chance this is bad dataless on import, just noting it for future reference.

#4 Updated by Mary Templeton over 9 years ago

In looking at the dataless CO_HAW.dataless generated from the XML file CO_HAW.xml, I see (using od) that for CO.HAW.00.HH2 epoch 2013,287,16:00:00 to present, the stage 3 gain blockette (B058) should begin

058003503+1.00000E+00+1.00000E+0000

but instead reads (with bytes translated to char):
S*03+1.00000E+00+1.00000E+0000

While PDCC cleans up after this ok, it is causing stage 3 to be omitted from the database load. Seedsniff fingers this issue as well. This appears to originate in the program that translates the XML to dataless.

#5 Updated by Mary Templeton over 9 years ago

This problem turned out to be a channel comment handling bug in PDCC and not a problem in either the stationXML converter or the metadata loading software.

In retrospect, the dataless from the stationXML converter failed to load because there was no <Sensor><Type> in the original XML for state of health channels.
The stationXML converter translated this as a B033 lookup code of 0, which had no B033 entry, so the load failed. In an effort to fix the dataless, it was edited in
PDCC, but PDCC inserted one of the channel comments into the HH2 response cascade between stages 2 and 3. This channel comment also throws the error:

Exception in thread "AWT-EventQueue-0" uncaught FX exception: edu.iris.Fissures.seed.exception.ContainerException edu.iris.Fissures.seed.exception.ContainerException: ERROR: buffer entry to tag >059.CO.HAW.a.00HNZ.2013_287_16_00_00.0000< not found.

at this.volume.container.get(this.tag) ("file:/Users/met/src/PDCC-3.8/bin/edu/iris/pdcc/gui/SeedVolumeTreeCell.fx", Line 325)
    at getBlockette(iteration) ("file:/Users/met/src/PDCC-3.8/bin/edu/iris/pdcc/gui/SeedVolumeTreeCell.fx", Line 301)
    at selectionModel.selectedNode = this ("file:/Users/met/src/PDCC-3.8/bin/edu/iris/pdcc/gui/PdccTreeCell.fx", Line 34)
Caused by: edu.iris.Fissures.seed.exception.ContainerException: ERROR: buffer entry to tag >059.CO.HAW.a.00HNZ.2013_287_16_00_00.0000< not found.

in PDCC. This dataless was written out from PDCC, but truncated itself after stage 2 of the HH2 channel, causing the (missing) stage 0 sensitivity to be missing.
I will report PDCC's channel comment handling issue to Rob Casey.

I removed the state of health channel information from the original XML file, Ellen Yu ran it through the stationXML converter and the resulting dataless loaded
in its entirety without incident.

#6 Updated by Chad Trabant about 9 years ago

Can this be closed? Doesn't sound like there is anything for the converter to do here.

#7 Updated by Philip Crotwell about 9 years ago

Yep, I think it can be closed. Just have to be very careful with PDCC and comments!

#8 Updated by Chad Trabant about 9 years ago

  • Status changed from New to Closed

Not a converter issue.

Also available in: Atom PDF