hckrnws
Oh wow, I made this! I didn't expect to see my map on here
Let me know if you have any questions or feedback and I'll do my best to respond to them.
Tech Stack used? Any specific non-technical difficulty getting this done ?
Tech stack is a mix of things, but the big-ticket items:
- Dagster for data pipeline
- Parquet files for data format
- Cloudflare R2 for data storage
- DuckDB for data processing
- NodeJS for API
- NextJS + React for the app
- MapLibre for the base map functionality
- Deck.gl for the complex map layer (eg. wind particles)
- WeatherLayers (+ GFS) for the wind particles
- Cloudflare as a caching layer
The main difficulty was learning about the underlying data and industry nuances. I've been working on this map the past few years and I'm still learning new things that force me to change my approach. It's an incredibly complex domain and it's part of the reason I made this, to try and make this complex data a bit more digestible.
It's a beautiful piece of work, thank you very much.
It would be amazing if this was extensible and wind farms and solar farms outside of GB could contribute their data as well.
Thanks! I would love to make this something that works beyond GB though right now that would require a fundamental change to the way things are set up. It's on my list though, so one day.
I do some volunteer work with the https://electricitymaps.com/ folks, highly recommend getting in touch if their API(s) would be of use. I can provide a contact if needed.
This has just had a major update to include other renewables (eg hydro, solar) as well as non-renewable energy sources (gas).
It’s also added grid boundaries and cabling recently - and I think there’s also wind roses planned?
My one UI wish - the energy sources filters are currently displayed as options on the map - rather than controls for the entire UI - which meant I missed them entirely at first. Would be great if they could be given more prominence.
Definitely wind roses planned, I posted about them on my LinkedIn page if anyone is curious (linked on the main map).
That's a great point about the controls. Originally the map only showed wind farms and as I've expanded the features the control system (and really the entire UX) hasn't kept up. I need to sit down and have a proper think about how best to design the controls and documentation.
FAQ also mentions it's only wind, so took me a while to notice :)
Would be nice to have a definition of 'Turn-up' and 'Curtailment' in that FAQ for us non grid savvy visitors.
Works great though, well done.
Ah yeah the FAQ is very out of date and in need of a full rewrite. I'll get to that eventually. And you're right, I'll make sure to add some sort of glossary and perhaps also explain things better in-situ.
It's pretty windy here today! Around half of the UK's generation is currently coming from wind[1]
> New wind power records are set regularly, and between 5:30pm and 6:00pm on 5th December 2025 British wind farms averaged a record 23.94GW of generation.
Cool!
Semi-related: I also have a page where you can track the wind records as they happen
Hey Robin, big fan of this map, congrats on getting front page on HN :).
Two suggestions/questions if you may:
1) Would be good to see how many MW each boundary can handle, not just %? Btw, I can't see the number for the south east england boundary.
2) Great job on the battery info. I'm seeing some battery storage is curtailed. How is that possible? Please don't tell me that we are paying batteries to _not_ export :/?
Hey! And thank you
1) Absolutely agree. The current approach for the boundaries is a quick hack until I can implement something more sophisticated. Safe to say an update is already in the works that adds a MW value and more insight into the state of each boundary (and is also more accurate in general)
2) "Please don't tell me that we are paying batteries to _not_ export" – it's actually the opposite, the batteries paid to not export (at least today). You can dig into this yourself via the Detailed System Prices dataset [0] and looking at one of the batteries on the sell stack (eg. KILSB-5)
> "Please don't tell me that we are paying batteries to _not_ export" – it's actually the opposite, the batteries paid to not export (at least today).
Unfortunately you've got at least one negative wrong in this sentence and I'm still confused, and the linked dataset is currently blank?
Sorry for complaining, this is a great website!
No worries. The Detailed System Prices dataset is lagged by a couple hours so try going back in time.
The simplest answer I can give is that assets place bids and a volume of energy that they are willing to turn down if the system operator needs to. Those bids are either positive or negative in value and this depends a lot on the type of asset, for example wind assets usually bid negative (ie. we pay them to turn down) while gas assets usually bid positive (ie. they pay us to turn down). The reason for that is a lot to do with complexities of the market and also the cost of running that assets, the cost of fuels, etc.
Ok, got it I think.
So actually the battery operator is paying _the grid_ to turn down output from whatever was previously agreed (because they think they'd get more money for it later?).
And this shows as curtailment on the map?
Let me know if I'm directionally right here. If I am, it would be good to see 'bad curtailment' vs 'good curtailment' (i assume if bids are negative/positive?)
Love this! Would be super interested in any details the author could share on the data engineering needed to make this work. The vis is super impressive but I suspect the data is the harder thing to get working.
The most time and energy has been getting my head around the source data [0] and industry-specific nuances.
In terms of stack I have a self-hosted Dagster [1] data pipeline that periodically dumps the data onto Cloudflare R2 as parquet files. I then have a self-hosted NodeJS API that uses DuckDB to crunch the raw data and output everything you see on the map.
[0] Mostly from https://bmrs.elexon.co.uk/ [1] https://dagster.io/
[dead]
This is really great. It’s fab to see this data being presented so clearly.
Been following your projects and writing since the Mozilla days, always impressed!
Oh wow that's a blast from the past, feels like a lifetime ago! And thank you.
What are the black dots around the map when you zoom in? For instance South-East Wales / Bristol, I see a few black dots that look like they cover shaded areas or something. They also show up at the sites for wind farms, possibly they're the turbines themselves?
Yup, that's exactly it! When you zoom in you get to see the wind farms and wind turbines using data from the amazing Open Infrastructure Map [0]. I also show the cables for the offshore wind farms.
Very cool. Thanks
Are those gas plants not peakers? It seems wild to me there is 5gw of curtailment while there is 16gw of gas going. Surely the grid could use some more storage capacity.
The curtailed windfarms are in the north of Scotland. There is currently not enough transmission capacity to export electricity from there to the south of England where the demand is highest. New transmission lines are under construction, but they wont be ready until 2029.
Virtually none of the gas power in the UK is traditional "peakers" aka Open Cycle Gas Turbines. There's a lot of Combined Cycle Gas Turbines. Gas is much more expensive in the UK than, say, the US and so it doesn't make much sense to build OCGT because although they're cheap to build they are very inefficient so your bid prices will be really high.
It might be that the grid lacks the capacity to transmit the power from renewables. i think I read somewhere recently that there is an argument going on about adding or upgrading connections in East Anglia.
Just noticed the wind farms even have the individual units ( I assume ) and their interconnections mapped?!
Would show that off and zoom into them when selecting them from the list view!
That's correct, it's using data from Open Infrastructure Map which is itself based on OpenStreetMap data. That's a good idea to zoom in further to show them off a bit more.
Nice job dodging the awkwardness of whether to include Ireland, Northern Ireland, etc (IE and NI operate as a single, all-island wholesale electricity market). :)
I actually have plans to include Ireland and Northern Ireland once I get the GB side nailed down. The data seems to be mostly available though I'd have to really think about how to make it work well on the app as it'd be a pretty fundamental change to how things currently work (assumes a single market).
Great visualisation...but doesn't seem to have any display of the amount of Solar (though won't be much on such a cloudy day!)
Some solar power assets are being tracked though unfortunately (for now) I'm only able to display data for assets that report on the balancing system in Great Britain. This means most solar has no data that I can easily access.
I hope to work around this soon.
Can someone explain what the lines mean?
The squiggly ones across the country? Those are the National Energy System Operator (NESO) transmission boundaries for Great Britain. They represent areas of the country that are tracked for the purposes of managing constraints, which is when the transmission network over the boundary can't handle the amount of power flowing over it. When they are overloaded (or rather, before that happens) NESO will step in and "balance" the various zones by turning up or down generators.
I note that one of those lines seems to take a big detour up and round Cruachan?
The lines are sort of arbitrary in a geographic sense as in reality they are defined based on the intersection with the schematic version of the transmission network [0]. Though yeah, that specific example is an odd one and I'd be interested to understand more about why it was placed in one boundary and not the other.
[0] https://www.neso.energy/data-portal/day-ahead-constraint-flo...
Lovely site. As a wind energy ignoramus, it would be great to add descriptions to your FAQ for the red lines, and curtailment.
I'll make sure to get the about page updated soon, it's been a bit neglected with being so focussed on the recent upgrades. Definitely a lot I can explain better, and also ways that I can solve that with better UX in general.
Great work, this is so cool to see!
Thank you
Really nice project :)
Thanks!
Very well done, beautiful data communication.
Yup! I have two ways to do it right now, the first is to click the plug-looking button on the map and this'll switch into curtailment mode to filter the assets that are currently being turned down due to transmission constraints, as well as the assets being turned-up elsewhere to balance things out. Also if you click on a curtailed assets you'll see the curtailment as a yellow area on the charts.
The other method is to visit my dedicated page [0] for tracking aggregate curtailment and turn-up, plus the costs for this.
Thanks for answering my curtailment question! I removed it from my comment as I commented before I found the data in your UX, but the extra context is useful. So much curtailment, much more transmission and battery storage required.
[flagged]
Do you approach coal power stations or oil rigs with the same level of scrutiny? I'll wait to hear back from you on the long list of environmental damage they cause. In your own time.
Crafted by Rajat
Source Code