Fifteen million stars in a browser

Cosmic Atlas started in March as a game. I wanted to fly a little ship around a map of the real Solar System. But I got more interested in building the map than in building the game, so the ship ended up parked in a branch called game/ship-prototype and I haven’t touched it in months. The map is now an atlas with around fifteen million real objects in it.

only real data

I didn’t want the atlas to show anything that wasn’t real. I didn’t like when any astronomical software would show made up stuff like star dust or procedural filaments. Yeah, it looks great, but it’s not real. I wanted this to be realistic. If there wasn’t anything in that part of the map, then so be it. Maybe that would inspire a scientist to chart it?

what’s in the map

Everything is drawn to scale around the Sun, and zooming only changes how much space fits in a pixel. I never compress distances to make the map look nicer, so the map is mostly empty space.

loading fifteen million points

At first I just put everything in Postgres and queried whatever the viewport was showing. It worked fine until I zoomed out and the viewport had five million stars in it.

So I moved the big layers to static tiles, the same idea map applications use. A build task cuts each catalog into a pyramid of tiles and uploads it all to a CDN. The tiles never change once published, they’re versioned (we’re on v2 now), and deploying only changes which version the app points at. Staging and production share the same tiles, the data is exactly the same anyway.

Postgres still handles the search by name. It also finds which object is nearest to a click, so the points you see on screen can turn into something you can open and read.

There’s also a check in the import script that I like a lot. It won’t consider production healthy unless it counts at least 1,597,012 stars in one layer and 1,339,910 in the other. Those are the exact numbers from the current import. If a future one counts less than that, it stops and I go check what happened.

getting the data in

The star data comes from Gaia, a European space telescope that measured the positions of almost two billion stars. The full archive is really big, so the importer runs on a workstation with a lot of disk. It verifies the checksum of every file it downloads, keeps only the nine data fields the atlas needs, and deletes the rest as it goes. It can resume where it stopped, because a download that big will get interrupted no matter what. One of the libraries refused to work on my Linux setup, and I lost an evening figuring that out.

The distant galaxies come from DESI, an American survey. That one is a single huge file. Its checksum gets verified before anything else runs, and I built the import so that running it twice produces exactly the same result, byte by byte. I wanted to be able to audit a release by rebuilding it. There’s a script that just does that check.

the moons of jupiter

Planet positions are calculated live from NASA data that the app downloads the first time it runs. For the moons of Jupiter and Saturn that data is huge, so the app asks a NASA API for their positions instead. During development I realized every page reload was hitting that API with the same request, so now the answers get cached and “now” is rounded to five-minute intervals. If you ask for an explicit date and time, you get that exact second.

what’s approximate

The positions of asteroids are calculated in a simplified way. That’s good enough to find them on the map, but you couldn’t plan a space mission with it. The distances to faraway galaxies are estimates. The sizes of stars are estimates too, based on their brightness and color. And the map is 2D, so depth exists in the data but not on the screen. Every object’s detail panel says which of these applies to it.

That’s all. The atlas is at skychart.org and the code is on GitHub. If you try it and something looks wrong, or an object you care about is missing, please let me know, I’d love to hear about it.