Project

General

Profile

Feature #541

Status update for PH5

Added by Rob Newman over 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
06/16/2014
Due date:
% Done:

80%

Estimated time:
Spent time:
Resolution:

Description

Hi Derrick,

Thanks for you attentive responses to the questions we have had. We
would like to start testing the ph5 interface more comprehensively,
and here are the things we would like to see. I have split them into
three groups for simplicity:

  1. feature requests,
  2. bugs, and
  3. APIs

(1) Feature Requests:

  • Is there the ability to have a top-level web page that has a map of all the ph5 experiments that allowed a user to click on to get to the form for an experiment?
  • Is there an entry point (semantic URL) to get straight to an experiment number, like we have with the current interface, eg: http://www.iris.edu/data/ph5?EXP=13-004
  • Can the display of what experiment (nickname, report num) you are requesting be passed to the request form and data selection tabs?
  • Can emails that go out to the user be sent more often (cron run every minute)?
  • We need notification when request fails. Please send to , for now.

(2) Bugs:

  • Email sent to our stats program (currently send to Sandy) has empty .LABEL line
  • Type of request needs set field as PH5 in the email we receive to interface with our request tables (archreq, etc)
  • Why doesn't the map (Google Earth) zoom in to where the stations are located (i.e. use the station lat/lon to create a map scaled to the bounding box)?
  • When restricted data is requested, the extraction happens even if the user doesn’t have access. An email is received later denying access. Can the check on restriction be done and displayed immediately in the web page without an
    extraction if the user doesn’t have access.

(3) APIs:

  • Documentation on how to invoke the standardized FDSN web services calls to return shot point locations and receiver stations in StationXML format
  • An example URL that does not require a login and shows how the FDSN web services are exposed.

Let us know if you have any questions about this.

Please let us know when new software is in place and what has been fixed.

Warm regards,
- Rob N

CC: Tim, Rick, Sandy, Mea

History

#1 Updated by Rob Newman over 6 years ago

  • Description updated (diff)

#2 Updated by Rob Newman over 6 years ago

Begin forwarded message:

From: Derick Hess <>
Subject: Re: ph5 interface issues
Date: May 21, 2014 at 2:26:53 PM PDT
To: Rob Newman <>
Cc: Tim Ahern <>, Rick Benson <>,
Sandra Stromme <>, "Mea Edmunds <>."
<>, Bruce Beaudoin <>

Most of the issues have been fixed and another update is coming on Friday. Below is a list of
bug fixes/added features.

  • A new way to get directly to request forms. Example: http://www.iris.edu/pic-ph5/cgi-bin/ph5?EXP=07-023
  • Data processing/email script now runs every minute
  • Every night the system checks for any updates to license files or experiment metadata and updates the web form accordingly
  • Map now auto zooms to an experiment using google earth. (A few bugs to work out tonight, but it works on most of the experiments)
  • Experiment name and report number now visible form any tab in the form
  • System now checks license at time of form submission before processing data and denies/accepts the request.
  • Stats email now properly writes .LABEL field and .REQ_TYPE PH5 added to this email. (These are currently set to send to )

Coming in the next update:

  • Password protected drop box area for PIC data group to add new experiments, with auto process and notification system of new experiment
  • More robust error notification system. Detailed emails about the suspected problem sent to key people
  • Web services will go up for testing and documentation for these will be posted

~Derick Hess

#3 Updated by Rob Newman over 6 years ago

On 2014-06-03 16:41, Mea wrote:

Hi Derick,
A couple of things about your comments below.

- There are current two lists to see what ph5 data sets are available.
http://www.iris.edu/data/ph5
and
http://www.iris.edu/pic-ph5/experiment_list.php
The list at
http://www.iris.edu/data/ph5
is useful and contains links which describe the data, etc.

The other list is for ???

- Nicknames should have some character limit, just for keeping lists each to read if nothing else.
For example, KPRPKoreanPeninuslaRefractionProfile should be just KPRP
They don't have to be that short, but I would hope that the name could be limited to 15 characters.
It is a "short name".

- I will need to know when new data comes on line for availability (restricted or open). How will
this be accomplished? An email to me, will allow me to notify all
the dmc staff involved in creating license files, updating database status, etc.

thanks
Mea

#4 Updated by Rob Newman over 6 years ago

Begin forwarded message:

From: Derick Hess <>
Subject: Re: ph5 interface issues
Date: June 3, 2014 at 9:48:01 PM PDT
To: Mea <>
Cc: Rob Newman <>, Tim Ahern <>,
Rick Benson <>, Sandra Stromme <>

I did write a new piece of code to access the forms via links like in http://www.iris.edu/data/ph5.
That link contains information that is more useful than what is auto generated by my software but
isn't found in the metadata. It might be best to to combine the two and use
http://www.iris.edu/data/ph5
but have the links go to the new forms. That will also help with the second problem which is the
nicknames. The autoparser just reads the nickname line in the ph5 metadata, so weird things show up
like the KPRPKoreanPeninuslaRefractionProfile. I believe I have write access to that data so I can
start fixing those nickname entries as well.

I just finished testing the new uploading form PIC will use and I'll have it send you an email when
new data arrives. When new experiments are uploaded they are processed and moved into the correct
directories, but no license file is created. This makes it so it is automatically restricted until a
license file is created on your end. Is there anything in particular you want the email to say or a
format?
Right now it has the report number, the person from PIC who uploaded it and their email address. I can
also include file size, experiment nickname, and PI if you need it.

~Derick

#5 Updated by Rob Newman over 6 years ago

On 2014-06-04 08:34, Sandra Stromme wrote:

HI,

Stats email now properly writes .LABEL field and .REQ_TYPE PH5 added to this email. (These are
currently set to send to )

Sounds great. We're ready to accept this email for our data distribution stats into a standard place.
Can you send it to , as well as ?

Every night the system checks for any updates to license files or experiment metadata and updates the
web form accordingly

Others need to decide this, but we often get requests for data from people who may have accidentally
been left out of a license file, and they want data pronto. Or, oops what if an experiment is
erroneously open (this has never happened), and we need to restrict it pronto. Is there a way for
us to goose this update?

Thanks.

Sandy

#6 Updated by Rob Newman over 6 years ago

Begin forwarded message:

From: Derick Hess <>
Subject: Re: ph5 interface issues
Date: June 4, 2014 at 7:46:41 AM PDT
To: Sandra Stromme <>
Cc: Rob Newman <>, Tim Ahern <>,
Rick Benson <>, "Mea Edmunds <>."
<>, Bruce Beaudoin <>, Anh Ngo <>

The cron script is small and fast enough I can run it as often as every minute if needed. On the
pic-ph5 machine you can also just run the /home/PIC/k3/www/restricted_cron.py if needed instantly.
Adding a new email to the license file doesn't require the script to run. Only a change in restriction
status. You can add a new email to a license and it will instantly be valid.

I'll add assembled_request@iris.washington.edu to the email.

#7 Updated by Derick Hess about 6 years ago

  • % Done changed from 0 to 80

Dropbox area for PIC now up on the server. Allows for pause and resume of uploads for large uploads. Turning on the email feature today to notify DMC (Mea as of now) and myself when a new experiment is uploaded.
Emails for potential errors added.
Web services up with exception of station xml format

Currently working on:
writing a new metadata to xml parser to be compliant with station xml.

#8 Updated by Rob Newman about 6 years ago

Hi Derrick,

Can you please add to the list of email notifications?

Thanks,
- Rob N

#9 Updated by Rob Newman about 6 years ago

Hi Derrick,

You wrote in your comment:

Web services up with exception of station xml format

Can you please give an example of how this can be used? (including a URL)

Also, how close is this for me to pass to Mea for another round of testing?

Thanks,
- Rob N

#10 Updated by Derick Hess about 6 years ago

There are 2 web services: Kef and metadata. For most users metadata will probably be the most useful. The kef web service is most useful for someone updating or working directly on PH5 files or writing software for that purpose


The kef web service base url is: http://www.iris.edu/pic-ph5/cgi-bin/kef/1/ and has 4 supported methods (about, version, application, and query).

about: returns basic information about the web service in plain text. It can be accessed at http://www.iris.edu/pic-ph5/cgi-bin/kef/1/about

version: returns the version number of the web service build. It can be accessed at http://www.iris.edu/pic-ph5/cgi-bin/kef/1/version

application: returns a WADL in xml format. It can be accessed at http://www.iris.edu/pic-ph5/cgi-bin/kef/1/application

query: This is the main application and will return various PH5 data in plain text kef format. It has 2 required parameters(exp_id and kef_type) as well as 2 additional parameters used in certain cases. examples below:

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=experiment_t
This will return all the basic information about the experiment 07-012

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=event_t
This will return the metadata for all events in kef format for experiment 07-012

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=array_t&array=001
This requires an additional parameter(array) to indicate what array number you would like. This request will return all the station metadata for a given array

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=response_t
This will return the response table metadata in kef format for experiment 07-012

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=Receiver_t
This will return all metadata in the receiver table in kef format for experiment 07-012

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=index_t
This will return all metadata in the index table in kef format for experiment 07-012

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=das_t&das=11979
This requires an additional parameter(das) to indicate what das you would like. This will return all metadata about a given das in kef format for experiment 07-012

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=time_t
This will return all metadata in the time table in kef format for experiment 07-012

http://www.iris.edu/pic-ph5/cgi-bin/kef/1/query?exp_id=07-012&kef_type=master
will return the entire master.ph5 file in ph5 format for experiment 07-012
--------------------------------------------------------------------------------

The metadata web service base url is: http://www.iris.edu/pic-ph5/cgi-bin/metdata/1/ and has 4 supported methods (about, version, application, and query).

about: returns basic information about the web service in plain text. It can be accessed at http://www.iris.edu/pic-ph5/cgi-bin/metadata/1/about

version: returns the version number of the web service build. It can be accessed at http://www.iris.edu/pic-ph5/cgi-bin/metadata/1/version

application: returns a WADL in xml format. It can be accessed at http://www.iris.edu/pic-ph5/cgi-bin/metadata/1/application

query: This is the main application and will return various PH5 data in either JSON or xml (station xml coming soon). It has 3 required parameters(exp_id , format, type). Examples below:

http://www.iris.edu/pic-ph5/cgi-bin/metadata/1/query?exp_id=07-012&format=JSON&type=Experiment
This will return metadata data about the basic experiment information in JSON format for 07-012. Changing format to XML will return the metadata in a basic xml format.

http://www.iris.edu/pic-ph5/cgi-bin/metadata/1/query?exp_id=07-012&format=JSON&type=Event
This will return metadata for all events in JSON format for 07-012. Changing format to XML will return the metadata in a basic xml format.

http://www.iris.edu/pic-ph5/cgi-bin/metadata/1/query?exp_id=07-012&format=JSON&type=Event
This will return metadata for all DAS in JSON format for 07-012. Changing format to XML will return the metadata in a basic xml format.

http://www.iris.edu/pic-ph5/cgi-bin/metadata/1/query?exp_id=07-012&format=JSON&type=array
This will return metadata for all arrays in JSON format for 07-012. Changing format to XML will return the metadata in a basic xml format.

----------------=--------------------------

I shoudl have new version ready for testing either tomorrow or friday.

#11 Updated by Rob Newman about 6 years ago

Great - thanks! Let us know when Mea and I can start testing.

#12 Updated by Derick Hess about 6 years ago

  • Priority changed from Normal to High

I have completed the new back end error handling system and need to install that. We have also found a few bugs in the kitchen software code that lead to failed processing of PH5 files. Steve is fixing those in house here and we will be updating the code on pic-ph5 as well. We plan on having everything ready for final testing within the week.

#13 Updated by Rob Newman about 6 years ago

Hi Derrick,

Can you please identify the bugs you have found by creating tickets on this system and assigning them to Steve. That is kind of the point of the ticketing system. All of us can then see the progress in real time. You should create individual tickets for individual bugs and make them related to this ticket.

Thanks,
- Rob N

#14 Updated by Rob Newman about 6 years ago

Hi Derrick,

Can you please give us an update? Your last note had promised everything ready for final testing 'within the week'.

Thanks,
- Rob N

#15 Updated by Derick Hess about 6 years ago

Steve will be updating the pn3 code on PIC-PH5 sometime today. This is code for handling/and generating PH5 files. I'm not sure what bugs he fixed in that code but he said he found a few last week. He fixed them and was testing on some data sets over the weekend. Once he has that updated I will be able to bring the system back up. Look for it to be fully ready for final testing tomorrow.

There are some low priority style and visual things Mea and I have discussed changing to make the user experience a bit better that will continue to be worked on but wont affect up time or ability do deliver data to users.

#16 Updated by Rob Newman about 6 years ago

Hi Derrick,

Do you have an update on this ticket?

Thanks,
- Rob N

#17 Updated by Derick Hess about 6 years ago

Steve updated his code on pic-ph5. I've run a few tests and it seems to be working fine now.

#18 Updated by Mea Edmunds about 6 years ago

Derick
Thanks for update. I'll run some tests for the problems listed on this ticket
and let you know if those issues are resolved.

Mea

#19 Updated by Derick Hess over 5 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF