Spdb information and setup

nguy · September 25, 2020, 6:50pm

Separate components of Radx refer to the use of SPDB for profile information. But the links in the documentation are not present. Does the following from the TITAN documenation capture what is needed?

SPDB data base
The SPDB (Symbolic Product data base) format was developed for simple and efficient time-referenced storage of a wide range of non-gridded meteorological data. It is used in TITAN for almost all data which does not readily lend itself to storage in a regular grid.
Examples of data stored in SPDB are lightning data, aircraft tracks, aircraft measurements, gust fronts, surface station measurements and hydrological measurements.
SPDB stores its data in ‘day files’, two files per day. See page 5 for details on the file naming convention.
The .indx file contains information on how the individual data chunks are stored, as well as some special time-index arrays to speed up data access. All headers in the index file are stored out in big-endian byte ordering by SPDB.
The .data file contains the actual data chunks stored in the data base.The SPDB sub-system does not attempt to interpret the data in any way. It assumes that the data chunk is in big-endian format when passed to it, and it stores the data exactly as provided by the client.
Programming access to SPDB files is performed using the C++ API in the libs/Spdb library.

jcdehart · September 25, 2020, 9:16pm

I’m not an expert on the spdb requirements (my only experience is using them in my LROSE workflows to extract soundings from model output), but the file suffixes match the files I have on my computer. My educated guess based on the text is that yes, that is the correct documentation.

Are you trying to ingest actual sounding data or estimate profiles from a model?

At the very least, thank you for reminding me about this omission in the documentation. I’ve added it to my to-do list.

Jen

nguy · September 25, 2020, 11:43pm

Hi Jen,

Thanks for the message. I’d like to use model soundings. I have a workaround right now just remaking the parameters file every time. But this is an interesting method that would be nice to expore.

Nick

jcdehart · September 28, 2020, 4:32pm

What is your current workaround? And more generally, what is your current (or desired) process?

In terms of my own workflow, I’m usually working on just 1 or 2 radar locations at a time, so I edit the parameter files manually because I haven’t needed anything more complicated. I’ve used a mix of the RAP and GFS model output and then I extract the “soundings” with Grib2toMdv and Mdv2SoundingSpdb, both of which are in LROSE. This basic process is described in this tutorial, specifically the Obtain GFS data and Sounding sections.

Jen

nguy · September 28, 2020, 7:08pm

I have “templated” the parameters file and have a script that takes in a sounding and inserts the temp profile into the appropriate Tpf variable. A little book-keeping of the parameters file (moving into known location to reference) is needed to use this.

Ooh, thanks for pointing that tutorial out. I must have skimmed it! I’d just have to scale this to work with many radars, which seems plausible.

jcdehart · September 29, 2020, 4:15pm

Oh, I see! I’ve had good success with the model-derived soundings, though admittedly my freezing level is usually ~5 km and homogenous in space. I could see observed soundings being more reliable in other environments, though I would need to ask Mike about how to convert soundings into SPDB.

In my workflow I loop over several days, so I think writing a script to run the algorithm for many radars is definitely possible. LROSE can write to radar-specific subfolders. In the example below, I manually run my script for different radars, but LROSE writes the output from each step in a different folder for each radar (and additional subfolders for each date).

I use environment variables that are sourced before each run to set the radar name and Z-R, etc coefficients in text files so I don’t have to modify each parameter file when I’m using the same procedures for different radars. I could see a similar process working well for your needs (though you will probably code it up better than I do since I’m not a UNIX expert ha!).

Also, if you come across any issues/confusion in the tutorial, please let me know. They haven’t been fully QC’d by an independent user yet - and I only made them a couple weeks ago.