SMART-ISO is a stochastic, multiscale model of grid operations under development at PENSA, which is currently being built around the full PJM grid. The purpose of this website is to provide ongoing documentation of what we have accomplished, what we are working on and features that we hope to develop in the near future. (If you have been to this website before, be sure to hit your refresh button to ensure you have the latest version.)
We recently made a major revision to the modeling of unit commitment decisions during the day. The old logic solved a unit commitment each hour, planning an hour to two hours in advance. The new logic exactly matches the process used at PJM. The change appears to have significantly improved the modeling of LMPs, especially during the peak month of July. Click on the "calibration and validation" link below.
Overview of SMART-ISO
Calibration and validation (new! October 11, 2013)
Study of offshore wind (with U. Delaware) (new! July 20, 2013)
Features (current and planned)
A video of SMART-ISO
The SMART-ISO development team
Overview of SMART-ISO
SMART-ISO is intended to be a highly detailed simulator of the PJM power grid, with special care given to the flow of information and the handling of uncertainty. SMART-ISO is much more than a "stochastic unit commitment model" which has received so much attention in the literature. It will simulate day-ahead and hour-ahead commitment and generation decisons, in addition to real-time economic dispatch at 5-minute intervals. SMART-ISO is intended as a simulator for performing a wide range of policy studies. It might also be used as a test environment for models and algorithms that might be used in production. It is currently using the full PJM power grid (over 9,000 buses and 14,000 transmission lines in 2010).
A major feature of SMART-ISO will be the correct handling of different forms of uncertainty, in addition to the careful modeling of the dynamics of generators such as steam and gas turbines. We pay special attention to the difference between when we make a decision (which determines the information available) and the actionable time (when the decision would be implemented). We carefully distinguish between decisions in the future that will be implemented (e.g. day-ahead decisions on steam generation) and decisions that we use purely for planning purposes (such as day-ahead plans of natural gas combustion turbine).
It is the goal that our effort be completely transparent. We will be doing our best to carefully document what we are doing. Due to confidentiality of some of the data, we will not be able to share the model at this time, but we are hoping to provide public access to the model, perhaps through a web site.
We have been putting a lot of work into calibrating SMART-ISO against historical activities from PJM. As of this writing, we have two forms of calibration: matching the pattern of LMPs., and matching generation across different sources.
The latest version of the model, with the more accurate modeling of PJM's IT-SCED unit commitment model, appears to be reproducing the pattern of LMPs quite well.
Pattern of LMPs
We recently completed a major rewrite of the "hour ahead" unit commitment model, which appears to have produced a major improvement in the modeling of LMPs when compared against historical values. The new logic, shown immediately below, closely matches the PJM IT-SCED logic. IT-SCED is run 15 minutes before and after the hour. The 1:15 run, for example, models the system from 1:30 to 3:30. We look at when each generator is planned to be turned on in this interval, and subtract off the notification time (plus ramp up time). If this time falls within the interval 1:30 to 2:00, the generator is turned on.
The old logic would run a unit commitment model (for gas turbines) each hour. The run at 1pm would plan from 2pm to 3pm. Any generator that was turned on during this period would be committed. Notification times were ignored. The LMPs using this old logic are shown below the latest LMPs.
The results appear to show that the more accurate model produces more accurate LMPs. In particular, look at the latest results for July, and compare to the previous results for July. The more accurate model appears to significantly improve the pattern of LMPs when compared against history.
Actual vs. simulated LMPs using the new logic that closely matches PJM's IT-SCED process:
Actual vs simulated LMPs using the old hourly unit commitment models:
Our first comparison is the distribution of generation across different types of generation, including gas turbines, pumped hydro, a tiny amount of hydro, combined cycle, steam and nuclear. On the left is the historical behavior, derived from the actual generation from PJM but restricted to the generators which we are able to map to the grid (about 93 percent of the total).
On the right is the distribution of generation from the model, where now we are running enough generation to cover the entire (actual) load. The plot on the left is slightly lower because we were not able to map all of the generators to the grid.
Note the pattern of pumped hydro, which is charged at night (the bit of blue at night). SMART-ISO initially plans the pumped hydro using a 48 hour look-ahead model, but the actual pumped hydro comes the hour-ahead model, which required the use of approximation dynamic programming to capture the value of water in the reservoir (without this, the model would simply use all the water right away, rather than waiting until the peak period). The simulated activities are those produced by the economic dispatch model that is run every five minutes.
These graphis required about six months of work. Not perfect, but pretty close.
Study of off-shore wind for the mid-Atlantic region
We are in the midst of a DOE-funded project, joint with the University of Delaware, to study the effect of off-shore wind on the PJM grid. As of this writing (July 20, 2013), we have almost finished calibrating SMART-ISO (but not quite). The University of Delaware has designed a series of five buildout scenarios, with the largest providing peak power of almost 70GW (for comparison, the entire PJM grid averages close to 80 GW, with peaks of 140GW).
Below is actual wind for the five buildout scenarios, with wind coming from 29 blocks spanning Virginia up to New York. Note both the overall variability, and the lack of any apparent smoothing from the contribution across the entire region.
We anticipate running simulations of SMART-ISO at different buildout levels, with and without grid capacity constraints. We hope to have results by October, but only after a thorough review by a number of parties.
Features (current and planned)
At this stage, it would be accurate to say that all aspects of the model are still under development. However, a version of the model is running. We are currently undertaking a major code revision that will give us the following features:
Below, we list features that are in the model (even if they are still going through enhancements) and then list planned features which we anticipate will be running within the next 12 months.
Features planned for fall, 2012:
Features planned for 2013:
Video of SMART-ISO
To see an animation of SMART-ISO, click here (warning - this is a 50 meg file). The animation is generated using PILOTVIEW 5.1, a package developed at CASTLE Lab for animating complex, dynamic flows (originally developed for our work in transportation).
This animation is generated from a run of SMART-ISO that still does not have grid constraints in the day-ahead and hour-ahead policies. This is because problems with our grid model are preventing us from obtaining a feasible solution at the moment. We use an unconstrained grid, and then evaluate loads on the links relative to rated capacities, which is how some links can be over capacity. The simulation also uses some simulated wind (shown as green circles along the coast) which generate more energy than the coastal grid can handle. We hope to resolve the issues with the grid later this fall.
Rectangles depict flow over the network:
Light blue - Line is at less than 75 percent of capacity.
Darker blue - Line is between 75 and 100 percent of capacity.
Yellow - Line is between 100 and 125 percent of capacity
Red - Line is over 125 percent of capacity
Circles depict generation or loads
Yellow - Loads
Purple - Nuclear generation
Red - Gas turbine or combined cycle
Gray - Steam
Green - Wind
The grid - We use the 2010 PJM transmission grid. Though we can use the full grid, in general we include the 69kV transmission lines and higher. SMART-ISO makes it possible to streamline the grid by limiting it to higher level transmission lines. Algorithms are used to map nodes on lower level voltage lines to the closest bus on the reduced size network being modeled.
Generators - After starting with generators described by PJM bid data, we purchased a dataset from Ventyx in 2011 to get more reliable cost and engineering information. We have completed the process of merging the Ventyx data with the PJM network, and now feel that we have a complete and accurate set of data.
Although the Ventyx data included ramp rates, we found that they were not accurate for a model stepping in five-minute increments. Instead, we used actual generator data from PJM to estimate ramp rates at the five-minute level. Many of these ramp rates were much faster than those estimated by Ventyx (which averaged changes over an hour rather than five minutes). We were also able to capture behaviors such as seasonal variations in production (gas turbines are more productive in cold weather than hot weather).
Loads - We now have loads for every five minutes for all of 2010.
Grid capacity - We believe that the grid data is generally quite good, but we are currently having some problems finding feasible solutions. We will use our Pilotview diagonstic tool to identify and correct what appear to be problems with some specific links in the network.
The major components of the system include:
We now have a single module that is used to solve the day-ahead, hour-ahead and real-time components. The module adjusts to each setting through the inputs. For example, both the day-ahead and hour-ahead planning problems require running the model at a time t, to plan a window from t' to t''. For example, we may run the model simulating what we know at noon to plan generation tomorrow (midnight to midnight) in hourly increments. For the hour-ahead model, we might run the model at 2pm to plan the window 3-4pm in five minute increments. For the real-time economic dispatch (five minute horizon) we are just running the power flow model.
A major feature of SMART-ISO is its handling of uncertainty. Considerable attention has been given in the research community to the "stochastic unit commitment problem" where models use a set of scenarios to handle different events that may happen tomorrow (this algorithmic strategy is known as "stochastic programming"). Instead of a single integer program optimized for a single future, stochastic programming models optimize over multiple scenarios, often producing very large scale integer programming models that need the power of a supercomputer.
SMART-ISO is first and foremost a simulator, which simulates policies for making day-ahead, hour-ahead and real-time decisions. When a decision (such as day-ahead and hour-ahead) affects the future, we have to design a policy to handle the uncertainties in our forecasts. One goal of SMART-ISO will be to develop and compare different classes of policies for working with uncertainty. Stochastic programming is one class of lookahead policy. Since we wish to simulate these policies over many time periods, computational speed matters, and we will be looking for robust policies that are much easier to solve than the stochastic programming algorithms.
Separate from the design of a policy (an algorithmic challenge) is the modeling of uncertainty. If we think of a policy as a black box for making day-ahead, hour-ahead and real-time decisions, we then have to think about how to model both the dynamics of the physical system (e.g. ramp rates), as well as the dynamics of uncertain events such as wind, solar, loads, and network performance.
We will put considerable care into the proper modeling of forecasts of exogenous events such as wind, and the need to do advance planning, such as day-ahead planning of steam generation, planning of gas turbines over 5 to 60 minute horizons, in addition to different forms of demand response which might require advance notification of several hours to a few minutes (or nothing at all). For example, we are developing models of forecast error for wind which captures both amplitude (quantity) and temporal errors. Temporal errors arise when we forecast an increase/decrease in wind due to the arrival of a high or low pressure zone, but where the timing is wrong.
We anticipate modeling the following types of uncertainty:
Documentation is limited at this stage (this website is primarily a high-level summary). At this stage, the best documentation are two senior theses written in 2011-2012:
Kevin Kim, "A Stochastic Unit Commitment Model in the Presence of Offshore Wind Energy," Thesis, Department of Operations Research and Financial Engineering, Princeton, University, 2012. - Kevin's thesis focused on the day-ahead unit commitment model. He used a simple placeholder logic for the hour-ahead model. He used a hybrid lookahead policy with a cost-function approximation to produce robust policies for handling the uncertainty in wind forecasts.
Steven Chen, "Natural Gas Power Generation in the Presence of Wind: A mixed integer linear programming approach to the hour-ahead unit commitment problem," Thesis, Department of Operations Research and Financial Engineering, Princeton, University, 2012 - Steve's thesis focused on the hour-ahead unit commitment problem for natural gas generation. His model takes as given the day-ahead decisions for steam generation, but is allowed to make last-minute adjustments to run natural gas combustion turbine generators, as well as to take advantage of spinning reserve and the ability to make small adjustments to steam generation.
The SMART-ISO development team
SMART-ISO is being developed under the direction of Professor Warren Powell, director of PENSA. Below is the development team, but the model would not have been possible without the direct involvement of PJM, which provided a detailed dataset of their network, in addition to providing valuable advice.
Dr. Hugo Simao - Responsible for software development and calibration
Dr. Boris Defourny - Expertise in power flow, generation and energy markets
Dr. Marcos Filho - Developed and calibrated the real-time AC power flow
The first version of the model was largely written by undergraduates in the Department of Operations Research and Financial Engineering at Princeton, primarily as part of their senior thesis research, with significant additions made by teams of undergraduates working as summer interns at PENSA under the supervision of Hugo and Boris. Major contributors include:
Jessica Zhou - Jess wrote the very first version of the model for her senior thesis, including a full simulator with day-ahead and hour-ahead logic.
Kevin Kim - Kevin worked on the day-ahead unit commitment model as a summer intern and then for his senior thesis.
Steven Chen - Steve rewrote the hour-ahead model, introducing an integer program for planning natural gas generators in five-minute increments using a model that was solved hourly.
Daniel Dix - Daniel developed a powerful set of graphical utilities that helped with the mapping of the PJM load and generator data to the PJM grid.
Austin Wang - Austin started in the summer before his freshman year and provided significant assistance with the process of mapping generator and load data to the PJM grid.
Ashish Gupta - Ashish developed a powerful set of machine learning algorithms that played a central role in our ability to map PJM generator and load data to the PJM grid, which used independent node numbering systems.