Carl Doedens: Solar nowcasting with latent diffusion and XGBoost: assessing feature importance and evaluation in different environments
Carl Doedens [1,2], Yi Huang [1,2], Caroline Poulsen [3], Kelvin Say [1,4]
[1] School of Geography, Earth and Atmospheric Sciences, The University of Melbourne, Melbourne, VIC, Australia
[2] ARC Centre of Excellence for the Weather of the 21st Century, The University of Melbourne, Melbourne, VIC, Australia
[3] Bureau of Meteorology, Melbourne, VIC, Australia
[4] Melbourne Climate Futures, The University of Melbourne, Melbourne, VIC, Australia
Solar energy is one of the most important technologies for climate change mitigation, but its potential and adoption is limited by our ability to forecast the weather. High levels of solar uptake increase the sensitivity of the electricity system to weather variability, necessitating greater forecasting capabilities across timescales. In particular, traditional methods are not well suited for short-term forecasts, or “nowcasts”, especially for variables as dynamic as cloud cover and solar irradiance. Machine learning (ML) models have emerged as tools that can provide accurate nowcasts of surface solar irradiance, as they can rapidly ingest and respond to high-resolution observations, such as from geostationary satellites, in near real time. With the utility of these methods well established, we evaluate how ML models perform in different weather conditions around Sydney and assess how the training and choice of input data affects the models’ behaviour. The ML library XGBoost is used to train a suite of solar nowcasting models using a combination of model data and geostationary satellite observations from Himawari-8/9. Early results from XGBoost suggests that the meteorological environment around Sydney has the largest impact on forecast skill, with the choice of input data playing a secondary role. Insights from the XGBoost experiments are being used to guide the training and design of a latent diffusion model for solar nowcasting. The latent diffusion model will then undergo the same testing process to assess its forecasting skill for different weather conditions and how different features contribute to the forecasts.
Audience: anyone working on/interested in machine learning, forecasting, and/or renewable energy
Keywords: machine learning, diffusion, XGBoost, solar, nowcasting, feature importance, evaluation, verification