Visiting NSF NOIRLab for the first time – looking for pixels from telescopes

Hello,

I found a link to https://www.darkenergysurvey.org/the-des-project/data-access/ and wanted to look at a couple of images to see what kind of pixels and visualizations to expect.

The link is to “NOAA Portal” at http://archive1.dm.noao.edu/ which says you are in transition to NOIRLab

DarkEnergySurvey should get updated.  Actually, it would be a good idea if all references to it were found and updated.

“darkenergysurvey.org” has 9,380 entry points

“dark energy survey” has 219,000 entry points

“TheDesSurvey” has 7 entry points

“DArchives” is not a unique string.

DArchives” “energy” has 231,000 entry points

“Dark Energy Spectroscopic Instrument” has 32,700 entry points

You keep saying “400 scientists” but potentially this data could be used to train tens of thousands of the 1.92 Billion first time learners from 5-20 in the world right now.  Because I look at the whole Internet, with about 4.8 Billion people with some access, someone proudly saying they are sharing with 400 sounds rather faint.

If it were my group, I would make sure that every mention of these terms would be linked to a small number of core starting points by a simple floating context icon over every page that contains them. That can be a browser add on.  I am looking at different strategies and methods.  The main thing is to provide authoritative links, not partial and unlinked copies of things.  It should be an organic whole, sensitive in every part, continually improving, continuously aware and responsive.

I am a young person (I am actually 72 now) searching the web for “dark energy” and come across that page above.  I get to this “NOIRLab Astro Data Archive”.  There are words “Explore our petascale library of astronomical data” but it is not a link, just something they said.  I am not sure what “petascale library” means and I cannot hover it.  Another one of those sites where they expect you to know who they are and all the insider words.

API Access means more reading long documentation, specialized software tools, and a long delay getting to any “data”.  Skip that.

“Login” – but no register.  So somewhere I have to get permission, and then I can use that one.  Skip that.

What are these huge </> Q ? things.  How intrusive and busy and unneccessary.  They just make a giant button.  Probably someone working on a cell phone or touch pad.  Skip or ignore those.  Can’t turn them off.  How annoying.

Logos for NSF and NORILab, but no hovers, no links, nothing.  What a waste of screen space.  They should at least have titles “National Science Foundation and what part of the NSF and a brief description and links in a basic hoverbox.  NOIRLabs logo should be the floating context symbol.  And “About” is standard for those, don’t need a separate “About”. But that is all they show.  No hover on about, so I have to open a whole new tab just to glance at who runs this page.

ABOUT: Right Clicked and opened https://astroarchive.noirlab.edu/about/ LONG delay with blank screen then statistics. They don’t know how to right justify a column of numbers, and the stuff is just text so I can’t put it in a spreadsheet. They have no note field to let me add somethings. So it is not collaborative at all.  Just the usual “click, go to a whole new page”

As I suspected, copy and paste of the table to Excel gives garbage.  You guys try it and see how hard it is to use those numbers.  I might want to start my data analysis with at least finding and learning what all those arcane and useless (to me just now) Telescope names mean.

Remember, I am a young person (probably high school) somewhere in the world.  I already know how to program in several computer languages, have my own web page, write games and parsers.  The usual high school kid.  My younger brother and sister got taught these things in school, I had to learn on my own, and now they are ahead of me. These things like “bok23m – 90prime” mean nothing to me.  The table has no image sizes, dates, what part of the sky, what area of the sky.

All those strange names need hovers, or columns.  I should be able to query ANYTHING on the pages and get more information. And the things that come up, those should be hoverable, no artificial boundaries or limits, just because someone did not know or could not find it.  400 smart people and they can’t make two pages of background hovers on all the basics.  “Shame on you” (that is me the 72 year old speaking).

I right clicked and pasted the table as plain text, and summed the column.  Pretty thin, but you would be surprised that most tables on the internet don’t add up, or have “numbers” that are not numbers.

Look at this text blob: “This research uses services or data provided by the “Astro Data Archive” at “NSF’s National Optical-Infrared Astronomy Research Laboratory“. “NSF’s OIR Lab” is operated by the “Association of Universities for Research in Astronomy (AURA), Inc.” under a “cooperative agreement” with the “National Science Foundation.”

I just checked, the National Science Foundation is a US Government organization that sponsors American groups and individuals.  (My hypothetical young person is from South America and hasn’t memorized these things).  https://en.wikipedia.org/wiki/National_Science_Foundation I had to search for it myself.  The nsf.gov was too off-putting and does not explain.  Wikipedia is flat, has the crudest hovers, but is usually ok.

Web Search:  I don’t want to “search the web”.  I want to see the data. Why say “search the web”?  I have learned little so far, except there are 15 million images in large collections, not a single sample on the landing page.

All Holdings pull down only has “All Holdings” not very useful.

“Target via coordinates”.  Yeah, sure.  I have no idea where these telescope are, where they are pointed, how much of the sky they use.  IF ANY OF THEM HAVE DATA I can actually download.  I want pixels, not more ambiguous stumbling blocks.

Object name.  (Me: My young person does not know that m31 is an astronomical name. Even if they recognize it as such, they have not memorized its coordinates and extent in the sky. And certainly have not learn RA and Dec and Radius. (72 year old guy talking now — what units are you using???  Decimal degrees?  Parsecs, Z, some other distance scale.  Come on.  You are smarter than that. This multimillion dollar effort that NSF apparently paid for, and 400 smart people have worked on, and none of them bothered to help make the data the most useful and powerful tool for those one in a hundred thousands of kids who like quantitative science and mathematics and computing and engineering?  1.92 B/100,000 = 19,200.  And those are likely to affect and influence a half dozen people around them on average.  Say about 115,000 bare minimum. And the core ones are going to be the only ones in their class usually.  So that many teachers involved.  So about 20,000 adults too. And parents.  Not a small group.  And you 400 ignore all them and put a text only, laborious, ambiguous, and slow portal online? Shame on you.)

Pull downs.  For new users, put ALL the dimensions out where people can see them.  I tried Observation yp

I cannot carry my example person for these details.  Needless to say, they cannot see anything at a glance, so you force them to open the boxes which disappear, losing context.  Trying to get the lists out of this code is a real pain.  Those dimensions are core to this dataset and you have them so stuck away I cannot just copy and paste.  Horrible.

At this point, my head hurts and I see that all the effort went into building yet more “programming required” APIs.

If you want to build global communities of users and make data really accessible to everyone ( the Internet if for everyone ), then you have to expend effort to walk in their shoes.  And that means all countries, all languages, all ages, all backgrounds.  It is possible.  It is not hard, just tedious. And small compared to the cost of every person spending weeks or months or years just to see the things this rich group of 400 got to see.

I am not being facetious or diminishing what you all have done. But I see the needs of the world and this kind of thing is the usual.

I, personally, like noise.  I study it in every kind of instrument and in every facet of the universe. So I saw “large camera”, “share” “data” and I got momentarily excited that I might actually be able to get pixels from a few images to count the patterns and properties of the pixels.  I don’t care if they are stars or galaxies yet.  I hope to learn that later, once I see if the data, its core visualizers, and tools are found and organized.

I know some is noise in the sensor. Some is noise from the earth getting into the sensor. Some is atmosphere. Some interstellar, satellites, and light pollution.  But it starts with pixels.  In a database, or text file, not a proprietary image format that only a few ten thousand people in a very select community can afford to read. The API.  I doubt it has any Internet formats (I have to think of what tools people have, not what a few hundred thousand or a few million have or can afford).

Internet “sharing” sites have to work for everyone.  So text, json, javascript are the most likely and the most universally available.  It you are not going to share with everyone, you have to say, we only serve these few and say how many and why the rest are excluded.

Science starts with numbers and counting. Sharing starts with imagining all the details of how many others will interact with something you want to share.  Putting a giant, inaccessible lump of “data” in a file on the Internet is not sharing.

In July 1998 I was thinking about how I wanted to spend the rest of my life.  I registered TheInternetFoundation.Org to use to put my notes on the Internet.  I had already spent 30 years working with large human populations, setting up data and tools for groups to see the whole of things.  I am proud of the Famine Early Warning System (FEWS.net) because it shows that a small group of people can monitor and intervene in something as complex and large as “famine”. The famine process is not some vague thing. I set up the software and computers and data feeds, developed methods and trained the people, learning and sharing with them.  The famine proces is extremely precise and there is a lot of data and people. Same with “all sky surveys”, “astronomy”, “astrophysics”, “camera images”, “covid”, “global climate change”, “online education”, “the Internet”.

Anyway, I had already built hundreds of websites and reviewed thousands.  I had set up websites for groups and owned and helped with many domains. So when I got an email from Network Solutions asking me to contact a senior vice president at Network Solutions, about TheInternetFoundation.Org, I was curious.  I called and he told me that Al Gore had diverted the money that was supposed to pay for the Internet Foundation ($15 per year per domain, about $6 B per year now) to put Internet into neighborhood. The US Attorney General ruled that “taxation without representation” and the whole thing got cancelled.  He asked me why would I take on something that was that large and was supposed to have that much funding and resources?  I said, “Better one person, than none at all”.

So I spent 20 years just studying and experimenting, scanning and learning, testing ideas, and counting lots of things. Then I started contacting groups about small issues and topics. Tracing through what people are facing.  I cannot put it in a few sentences in this message.  I hope to share what I found. But some of it is the absolute certainty that “the Internet is for everyone” and “the first purpose of the Internet is the survival of the human species”.  It is an extension of the human species, its memory, its knowledge, an instantaneous guide for everyone.  Or should be.

The thing that caused my to start reaching out to people is GW170817.  I had already measured the speed of gravity using public data from the small but global network of superconducting gravimeters and repeated that with the broadband seismometers that were sensitive enough to track the sun and moon tidal signal. But GW170817 showed that the speed of light and the speed of gravity are identical, not just close, identical. And that meant, to me, that can only happen if they share the same underlying potential. You might not care about such things, but it excited me. And I knew that the people working in so many different places would not grow as a community, unless they had tools to work together. So that is part of why I spent so much time on geophysical, astrophysical, atmospheric and sensor issues on the Internet. Plus covid kicked many brick and mortar groups onto the web – unprepared, poorly trained, poorly equipped for global scale collaboration in real time, in infinite depth.

Richard K Collins, Director, The Internet Foundation