Curating Weather Simulation Data. Earthcube Workshop in North Dakota.

“Simulation outputs are important but that does not mean we save them forever” – Gretchen Mullendore

This week I have been attending a workshop on data curation (a key part of open science) specifically on developing guidelines on the data produced by weather and climate simulations. Open science is better science! But a blanket “you must save and provide all data” is not only onerous (especially for underserved institutions) but not what is needed for reproducibility and reusability.

So many great minds focused on open science.

First, this post are my thoughts and do not, necessarily, reflect the views of attendees and organizers. There will be a report. There has been a lot written about measurements and measurements can no be recreated. Model data, to a degree, can be regenerated. By sharing workflows those with the appropriate resources can run the models on provided initialization and configuration data. Furthermore the sharing of workflows allows the exploring of the robustness of conclusions to assumptions (sensitivity) and the reuse of the workflow to address new science questions.

Gretchen kicking off the meeting

I really enjoyed the discussions and applaud the team’s focus on designing rubrics as it brings the conversation up a level and enables the clear measurement of the efficacy of solutions. It was also great seeing a huge diversity in the career stage and “flavor” of participants. We had data creators, curators, representatives from three publishers (AGU, AMS and PLOS), data scientists and more!

Susan from the University of Michigan on data curation.

Also, fittingly, lots of discussions around equity. Open science is better science. Journals are increasingly requiring data to be made available (even FAIR) which can create a burden to institutions without the physical and/or workforce to meet these requirements. There have been discussions of carving out exceptions for underserved communities. My perception is that the community here at the workshop pushed back hard against that idea as, as aforementioned, open science is better science. Rather we need to equip those institutions to meet the open science requirements.

Lots of discussions on just how much data should be required to be made available to be open and how long it should be curated for. Again a focus on designing rubrics to guide the process. The focus should be on the goal and be flexible to aid the scientist in achieving open science and reproducibility and also allow the society driven journals in meeting the aspiration of is members.

A nice atmosphere and a nice atmosphere!

It was great to be back in Grand Forks. The University of North Dakota is a great institution that, in the atmospheric science, punches way about its weight. Two of our recent three hires had a background at UND and I very much enjoy my collaborations with the team there. It was also very nice to be there during a dry cool air outbreak in summer rather than a frigid cold air outbreak in october!

SciPy Thoughts

Subtitle, too busy to blog. Just about finished my time here at SciPy and I am both tired and energized. My excitement has not diminished from my first SciPy back in 2012. Great to meet new people and re-meet people that, due to reasons, many of them pandemic related, I have lost contact with.

Good to be back in the ballroom!

My number one take away from SciPy is: How much better organized the community is and how they, more so than any government program I have worked with, pull in the same direction and work in concert across many projects. The impact of organizations like Chan Zuckerberg is clear as is the orchestrating role of NumFocus. Also a thing to watch is the new Scientific Python organization which is aimed at sustainable growth and enhancement of the ecosystem.

Queso!!!

The increasing common language of enhancement projects (PEPs, SPECs, ZEPs etc…) and common governance structures is extremely pleasing and what just blows my mind is how this is completely self organized without any kind of edict from above.

The Scientific Python ecosystem is just that, an evolving ecosystem! It is so pleasing watching it evolve to a sustainable track. As Ben Blaiszik said during his keynote, this software is fundamental science infrastructure and while it needs (very much) more financial support from the agencies who’s science it supports (side eye at DOE) it is now in a place where any funds it (the ecosystem) receives will be used for the good of science.

On a technical note some great things I took home were: New, exciting 3D visualization tools, Pangeo forge forges ahead, cool ways to access HRRR as a X-Array like Zarr store from AWS, James Webb space telescope processing runs on SciPy, new ways to manage conda environments for teams and more.

BBQ and storms!

On a professional note, my greatest enjoyment was from seeing the enjoyment of my team three of whom were at their first in-person SciPy. Joe, Max and Bhupendra seemed to completely immerse themselves in the meeting and made new connections. It was also fantastic seeing our ARM collaborators at Brookhaven Lab , Die Wang and Sid Gupta there. This turned into a mini-science meeting as well with new connections made and new work planned. It also is a sign that open science is growing in the programs I love.

On a personal note, it was fun and a little interesting being in Austin during the pandemic. The city’s homeless problem has gotten worse and many businesses are struggling with hiring and some old haunts have gone out of business. I really enjoyed taking advantage of the scooter scheme clocking up 25 miles of low carbon transport.

Great seeing out DOE EESSD funded open science family grow at SciPy.

The news today of NumFocus taking over from Enthought as the organizing entity for SciPy is great news. Enthought has been spectacular and so supportive but having a genuine not for profit will help in many ways. It also opens the opportunity for SciPy not being in Austin. I am genuinely on the fence about this. Whatever the case I hope NumFocus takes a good look at WHY we have these meetings and comes up with some guiding principles. Define what is trying to be achieved, a north star to guide decisions. Then they and the chairs, committee, etc, can keep coming back to those and be forced to justify decisions. I am excited for the future, be it in Austin or elsewhere (note the contract for Austin in 2023 is signed, this does not mean it has to be in Austin but means there is a cost to not having it in Austin).

I’ll finish this blog post by asserting I need to become more engaged in the community. I need to write in folks like NumFocus, Quantsight, 2i2c et al into grant proposals as collaborators as not only are they better positioned to implement workflows I love to use funding them will give back to the tools I love to use. I also need to make more time to contribute code and continue to support my team in contributing to free open community software, critical international science infrastructure.

SciPy 2022. Kid In a Candy Store.

Short update! I am SciPy bound. My first in person conference since, well, the world stopped. So the pandemic is by no means over and there is some controversy (which I will not go into but you can GTS yourself) but that has not dampened by excitement.

My first SciPy. Red pill all the way.

One super exciting thing is three members of my team, Bhupendra, Max and Joe, are heading to their first SciPy.. I remember my first SciPy. It was like a scene from the matrix where I took the red pill and my world changed forever. I have been in “science” for two decades plus and I have never found a community like the Scientific Python community. The smartest and kindest people I have ever met. Genuine and passionate.

Great day for traveling.

I am excited to re-meet many I have met (please please forgive my memory for names, the pandemic has frazzled the skills I had, already meagre, in that area) meet new people and just learn a lot! I remember clearly in 2012, my first SciPy attending and about to give a talk and wondering what to use to format code (at SciPy they show a LOT of code, it is amazing). And I heard about this cool tool called an iPython Notebook. Yeah, before Jupyter.

And that is the amazing thing about SciPy. You are, as Hamilton would say, in the room where it happens. In the very least you are in the hallway outside the room and are the first to know about what happened and use the tools of said happening. Bring It On!

Thank You.

Goodbye Houston. IAH on climb out.

Day eight, a travel day back home. Very thankful to be heading home. It has been a good, long, good trip and very worthwhile. I have a better understanding of Houston meteorology, better understanding of ARM Operations which will help me be a better scientist working with ARM data and many new connections to the area and collaborations.

Daniel, a big U of H fan!

Of the seven days I was mission scientist we called three “up” days. Two (the first two) were a slam dunk. Yesterday’s was less so with storm firing more to the east and north of the TRACER domain. The data from yesterday will still be interesting in studying a transition to an atmosphere more conducive to storms.

David Oaks. Michicagian to Alaska, Norway and now Texas.

Mike Jensen, the overall PI (lead) for TRACER will now take over as the Mission Scientist while Chris Nowotarski from Texas A&M will take over from Bobby as forecast coordinator. It was really nice today to not be “on”. Not having to plan my day around spinning up and having the call.

Gabi Pessoa, from theater in Rio to balloons in Houston.

Most of all I am thankful to the AMF site staff. David Oaks, the technical lead, Daniel Bahrt, Mark Spychala and Gabi Pessoa. Not only are they key to collecting data that will change our understanding of our planet they do it with style and a passion for the mission that is amazing. It is interesting to think when I attended a break out meeting at an ARM-ASR meeting (I think it was in 2016) when folks were just starting to talk about Houston and then I decided that we (Bobby and I) should start studying Houston storms using data from the local NWS NEXRAD radar it would lead to this.

Mark Spychala. Valpo grad, Army and NEON nerd now geeking on Texas clouds.

Also amazing, when Mike led the proposal and we came up with ideas for deploying to Houston that our science thoughts would lead to the uprooting and redeploying of people. Fact is, as much as we try, ARM’s deployments can not be fully automated. In La Porte, where we are deployed to the middle of an airfield, launching sounding balloons takes careful coordination with the airport, authorities and Houston TRACON. And even beyond soundings each technician does daily rounds where they inspect instruments, clean windows and work with visiting scientists like myself.

What now? For one I will be back in August for another tour of duty. Details still to be determined. With the ARM C-Band radar in automated mode our team will have our work cut out for us working out exactly what we got. And that’s where the coming weeks I will be wearing my other hat. Not a Mission Scientist but an ARM Translator. Working with the team to make ARM radar data more useful to our users. Can’t wait to dig into that data and so grateful to the folks on-site who make it possible. I leave Houston thankful.

Out On A High Note.

And we are done. A seven day tour as mission scientist for ARM TRACER done! Seven days and three up days, super stoked with that, especially this last day which was just a bonus. Forecast turned distinctly more positive for the science we want to do yesterday (understanding the impact of those little particles on big clouds) so we called an up day for my last day in Houston.

Done… Celebratory beverage at St Arnolds

So now I sit here biting my fingernails waiting for storms! I’m sitting in the trailer at the ARM site hoping to get video of one last sounding launch. Plus some time lapse of I HOPE developing storms. It has been an amazing trip. I have learned a lot about Houston, the meteorology, ARM and AMF operations and the folks in Houston. Being immersed is important. Back in Chicago I would have so many distractions. Here I watch the atmosphere develop.

Leading up to the forecast call I am watching the data, watching the sky and visiting the sites. I listen to the forecast call, take in all information and make a call on operations. I then watch how it pans out. As talked about in previous posts, forecasting creates an impression on all those involved and aids in future analysis.

Magic

With today’s forecast being so uncertain a lot hinged on the morning (7am, 12 noon in Universal Coordinated Time or UTC aka Zulu or “Z” yeah.. a lot of odd stuff) sounding and how dry the air just above the surface is. So, because it seemed “right” I took Mark up on his offer to come out and launch the 12Z sonde.

What a treat. There was a fine layer of fog over the site and the light was MAGIC.

So lucky to work for such an amazing organization.

So, scratch “get up at 5am on Sunday to drive to a world class facility to launch a balloon to help in your forecast” off the bucket list. A high note indeed. We looked at the sounding and kept the “up” forecast in place. It’s 23:30Z right now (yeah a long day). Still feeling good about the call. Even if it is a miss it is an interesting miss and the weather turns messy tomorrow (yay! A travel day with tropical-like storms!). I am really looking forward to getting back to my family. 2.5 years or lockdown and heck yeah I miss them when I go away (took all of about 24 hours). Proud of the work I have done here and so very humble to be part of something so amazing.

Putting The M In ARM

ARM’s Cloud Radars
Total Sky Imager and sun photometer.

Atmospheric Radiation Measurement. A facility 30 years strong. And the focus is on measurements. We take measurements. ARM collects measurements with a mission to improve the representation of clouds, aerosols (those tiny particles) and anything that impacts how sunlight travels to the earth and how the infrared radiation emitted from the earth travels our into space in any kind of earth simulation. ARM does not do the improving of simulations.

We target our data so users can do that themselves and use the funding from various agencies (including our sister program, the Atmospheric Systems Research program). ARM looks to scientists like me (yes I have a dual role here, I am supported by the ARM program to value add ARM radar measurements AND I am a scientist who proposes to ARM to deploy instruments) to suggest through a proposal mechanism where we should go to make the most impactful measurements.

Micropulse LIDAR.

ARM measurements are comprehensive. A simple breakdown could be instruments that measure the properties and chemistry of tiny particles, instruments that measure clouds, instruments that measure both and those that measure the air (the meteorology) in which they reside.

Those vital balloons.

I could go over each instrument and its purpose but the would be a lecture, heck a whole course, not a blog post. Some of my favourites are our cloud radars (of course) which shoot radio waves (super high frequency. Your wifi is ~3GHz our Ka Band radar is 35GHz! ) the micropulse LIDAR which instead of shooting radio waves shoots laser beams and, one of my all time favorites due to its simplicity, the Total Sky Imager (TSI).

Warm work!

The TSI cam from the idea of “hey I can see the whole sky in this shiny salad bowl”. It is my got when I am trying to work out “what happened?”. And, of course, as you have all garnered from previous posts, one thing ARM does very well is launch weather balloons.

So after taking all these photos in the morning (sweaty business even at 8am) I relocated back to the hotel for the weather briefing and ops call. The forecast was improving but is a real head scratcher for tomorrow. So we are calling an up day for tomorrow. The key will be moisture near the surface. If there is just a slight bit more than forecast we will get some nice storms. If not enough, no storms until a major chance comes through tomorrow night. Fingers crossed all!

Important People, Important Work.

Day five of mission scientist duty for TRACER. And a Friday to boot! In addition to my normal duties I have offered to step in an guide a visitor to the site today. Jorge Gonzalez is a professor from City Colleges of New York and NYU at Albany and a member of an advisory committee to the area of DOE that manages ARM, BERAC.

Full hat and dayglow!

The best day for Prof Gonzalez’s visit was also the day we have major crane operations building the tower and lifting the radome for the CSU C-Band radar. So it was full safety and full safety gear. While the site staff are the experts I could offer a little science talk for the visitor. I love showing folks ARM sites as it always ignites my passion for the facility and the amazing depth of observations we take. No other agency comprehensively measures the atmosphere like ARM. From water coming out of the ground through evaporation, to those tiny particles and the chemicals that make them up to how water and ice clouds are layered in the sky.

Prof Gonzalez and Dr Rahman enjoying a tour.

And our data is free and open. That’s right, all these people laboring to take measurements and they don’t get to keep it to themselves. Thanks to the taxpayer we give it all away for scientists to do with as they will (we hope and make sure we take the right measurements to improve earth system simulations).

Jorge and postdoc Kalimur enjoyed their visit, albeit it a bit hot under those hard hats and departed to the University of Houston campus to launch their own weather balloons. I relocated to a lunch spot between the site and my hotel to listen in to the forecast call and lead the operations call. There was a little excitement as soundings showed the atmosphere to be moister and a little more unstable than expected.

A cocktail bar with screaming fast internet! Ya gotta work where you can work!

But not enough for those big clouds that have us so excited to study the impacts of little particles. So we decided to keep those valuable soundings (and more valuable helium) in reserve for a day with more scientifically interesting clouds. Sunday onwards is seeing a dramatic shift in weather patterns. Talk turns now to safety in case we get so much rain next week we get some flash flooding. Fingers crossed for amazing weather for science that is also non-damaging to Houston. Always the creed of the weather geek. Now, it’s friday night. I am missing my family terribly but it is time to unwind a bit. Kemah has a distinctly Louisiana feel (more Cajan, less Cowboy) so relaxing while writing this post with a cocktail or two in a funky place called the Voodoo hut. Here’s to the weekend and some weekend science all!

Texas, Fine Weather and Fine People

Clear still morning.

Today is a down day. A day to catch up on some tasks back in Chicago, a day to do some planning and a day long planned to pay a visit to our friends at the National Weather Service. The winds blowing moisture in from the Gulf of Mexico slackened overnight which meant for the first time in a few days now early morning rain clouds.

In fact it dawned clear! I headed down to the nearest coastline on Galveston Bay and just missed some Dolphins, but I could see a lot of scared fish still jumping out of the surface. Really happy with my choice of Kemah as a place to stay as it is so walkable.

We are running dry in Houston!

As you may have seen from my posts we have an active forecasting activity during what is called the “Intensive Operational Period” or IOP of TRACER. We forecast for a variety of reasons; It helps us target out limited (1 in 4 days during the IOP from June 1st to September 30th) days we can call for enhanced operations for soundings (weather stations attached to balloons) so it is essential to forecast the most scientifically interesting days to do so! Right now pickings are a bit slim. Which is why nailing the right days to go “up” is essential. Second, it creates a record of “what happened”. By becoming “operationally aware” and recording our forecast briefings we create a impression of the sequence of events which will aide in future analysis.

The best folks!

Finally; it is FUN! While this sounds flippant, fun matters. Having fun is important. Being fun brings in young scientists and engages the old hands in mentoring them. I was young once. Heck even Mike Jensen was young once and now he lead a massive field project. I am SURE a future leader is one of our forecasters or assistants on this project today.

Radar controller for the world famous KHGX radar!

To this end I am so grateful to the fine folks at the National Weather Service at the Houston Galveston office in League City, Texas. Having the team at the NWS office who really know their stuff is so vital. The calls we have each day are forecast discussions. The forecast team (forecast coordinator, three bench forecasters for verification, clouds and convection and air quality) and 1-2 assistants (most important job) come into the call presenting the current picture of the weather. Everyone on the call then discusses this and amends the thoughts of the team.

Having the NWS as the first “sounding board” is amazing as they add the local knowledge to a diverse ground of minds focused on Houston. It was a real pleasure to visit the office today. So happy to have these NOAA professionals involved in enhancing a DOE project aimed at improving our knowledge of those tiny particles and their impacts on big clouds to improve weather predictions. From tornadoes to climate!

Feeling Lucky

Second up day of my time in Houston as a mission scientist. Early storms were not as strong but this is a good thing as it really allowed the energy and moisture the big clouds we are studying to grow. After two days traveling all around Houston today was a day to stay closer to my home base in Kemah. An interesting little tourist area with a boardwalk and funpark I chose Kemah because it is very walkable. In an area like Kemah I can get out and roam and observe the clouds we are so interested in.

Panorama of morning showers from the Kemah boardwalk.

This morning I walked out to the boardwalk as some early storms were forming and moving into the area. The bay affords a great view back. The forecast from Bobby and team yesterday was spot on and storms formed on a later forming Gulf breeze (a type of wind generated by the differences in temperature between the waters in the Gulf and the land). Too successful in fact as one of the storm systems hit Liz’s drone site! But exciting to see how those big clouds impact profiles of temperatures.

A hotel TV makes a great tool for zoom calls!

Very lucky to have a nice coffee shop in Kemah. After a hot walk I eschewed my usual choice of a latte and got an iced coffee. After seeing to some tasks back in Chicago (being a department head means it is hard to leave my duties back in the Midwest..) I hopped on the forecast and operations call for ARM TRACER. We are very lucky to have a range of very talented forecasters from professors to VERY capable students and even NWS staff. After two very lucky days things are drying out in the TRACER region (~100km around La Porte) although some storms are possible to our north. Good news is storms are firing away today so we did not have to abort enhanced operations at the ARM site and we have collected another very nice case study to understand the impact of tiny particles on big clouds. However, as mission scientist I called a “down” day tomorrow.

Feeling lucky indeed!

Of course normal ARM operations still means unprecedented frequency and fidelity of observations Right when the forecast call ended the whole Kemah area experienced a power outage. A quick pack and I went mobile again finding a local pub with blazing fast internet. There I took another ARM call, this one on the use of AI on camera systems. It was actually (lucky!) I went mobile as on the way back from League City I was treated to a show of a developing big cloud that go so big it hit the top of the atmosphere and formed what we call an anvil for that classical thunderstorm shape. I was so happy that we were launching soundings and our friends from TAMU, OU, TTU, Stonybrook and other collaborators were out there with mobile radars and other platforms measuring that storm right over Houston! Lucky indeed.

How it started, and how it went!

Up And Up In The Air

Yesterday on the forecast and operations call as the mission scientist I called an “up day”. A day where the ARM Facility will place its observatory in a state of increased vigilance. More soundings (weather stations on balloons to understand how temperature, water and winds vary with height in our atmosphere) and our radar our in Pearland, Texas will execute a special algorithm to follow storm systems. This was based on a forecast produced by the TRACER forecast team who are doing an outstanding job!

Oli and Marcus read data from ARM’s Micro Pulse LIDAR.

I will not bury the lede, the forecast panned out! An amazing day of storms in the Houston region well captured by DOE and NSF radar systems! Now, for the Up In The Air bit. I have several missions here in Houston. One is to understand the wider TRACER field program. Today I connected with my long time friend Marcus Van Lier-Walqui. We both want to understand the Unmanned Aerial Vehicles (UAVs) part of the ASR funded component of the TRACER campaign. We headed south west of the ARM site featured yesterday to sites run by Colorado University in Boulder and the Cooperative Institute for Severe and High-Impact Weather Research and Operations (CIWRO) at Oklahoma University.

Justin heads into the fields of Texas to retrieve a UAV.

We started the day out with Gijs, who was here with Justin, Jonathan and Radiance. They were flying a very impressive UAV fixed wing platform over pasture here in Texas. The launch system was amazing and while completely safe was a little intimidating. Managed risk! this is what the best people do. A bungee cord rapidly accelerates the UAV to get lift so the rotor can take over thrust once clear of the launch platform (a table… your tax dollars being spent very wisely!). The platform then measures the winds (heat and water) that goes into those big clouds we are studying. The heat and water are vital to understand because we want to see clouds that have the same heat and water but different mixtures of tiny particles. Only then can we understand how those tiny particles change those big clouds.

Gijs mentoring and inspiring as always.

Our next stop was to visit Liz, Michelle and Francesca flying a completely different UAV, a quadcopter. This amazing platform, more akin to the platforms starting to dominate the consumer market, does a profile of the same heat and water for those big clouds from the ground to 2,000 feet every 15 minutes! To put that in perspective I am crazy excited about ARM launching those balloons every hour. Both platforms are posterchilder for American ingenuity, home built over a decade with the kinks worked out by trial and error. I feel so honored to be an intruder on work that has taken blood sweat and tears over many years to make work.

Michelle and Liz capturing vital data with the CIWRO Quad Copter.

Our final stop was back at the ARM Mobile Facility. Since we had requested enhanced soundings we wanted to check in. Of course the team had it all in hand and we even got to see the Colorado State University radar being installed. Even better, as we arrived the amazing staff at the AMF were launching one of our special sondes! Marcus’ son Olie was with us and Gabby offered Olie the chance to launch the Sonde! So, TRACER scientists, your 20:30 UTC sonde and the science from it is courtesy of Oli Van Lier-Walqui!