Facebook AllSky group – https://www.facebook.com/groups/172438633343696
I have found thousands of live web cameras on the Internet. Here is one that has stars often – https://www.youtube.com/watch?v=mb2ZzwShJog
I saved a screen print of the whole page when found good stars. Then I opened that in an image viewer and clipped the sky. I went to https://nova.astrometry.net/upload and uploaded that clip — setting the advanced settings to no down-sampling (1), and wide field (10-180 degrees). These bright stars are easy for them to find, so I got a plate solution very quickly. I am not familiar with their results, but it was clear and unambiguous. Now I can use that to calibrate and document that camera, its location, view and sky. I can use that to point Google Earth to the same view. Or to put data onto the image as an overlay.
The same method (a few minutes not very hard work, assuming you know what to do with the data once it it found), could be used on any image where there are stars or planets. Not sure if it works on moon and a few stars.
Anyone doing something similar? I can now read the video data and clean it up. I am trying to standardize all the cameras on the Internet, including allsky cameras and telescopes of all sorts.
Richard Collins, Director, The Internet Foundation
Here is the approximate view on GoogleEarth. I know the plate solution has the information needed to fit the two. Just have not had time today to do that.
https://earth.google.com/web/search/Popocat%C3%A9petl+Agosto/@19.03303161,-98.6215183,4739.79160408a,7354.39838844d,35y,160h,90t,0r/data=CigiJgokCWOGCDCTIkVAEXPFP8CbIEVAGXGETGnGPljAIZMNbkVKQFjA
I contacted that group to see if they would help to streamline their site a bit, and cooperate with identifying all “sky” images and videos on the Internet. Particularly those live cameras that everyone on the Internet can see. My complaint to them was they are using such complicated terms and methods it is hard for a well trained person with perfect memorization to use their site, let alone every young person (about 1,9 billion from 5-20 in the world now) and 4.8 billion Internet users. Not sure telling them they are only serving one in a hundred thousand of their real audience is wise, but I sometimes just lose patience.
If you know the time of an image, and the location and rough direction, the solution can go really fast. They use unsupervised search and often have their big computer search millions of regions and all the stars. But these are only the brightest stars showing, and only a few, and in more or less known times. So I probably will have to find those datasets again and try to do it. I really get tired of finding these groups who give the Internet sharing to the last person in. Rather than making it a first priority – to serve all people, not just a tiny few, and no exceptions.
Every image, every equation, ever number and unit, every constant, every dataset, every person, every bit of writing, every table, ever data stream, every video stream and all its pieces – each thing on the Internet should be exactly identifiable. Then when a new person comes to the Internet, they will see everything neatly identified and ALL parts traceable. Rather than the horrible pile of unlabeled and untraceable stuff that makes up the Internet now. The only similes I found for how the Internet is organized – it is not organized, like the slums around large cities, like a wall of graffiti, like the libraries tore out all the pages of every book and magazine and threw then on the floor in piles, like a termite mound made by blind termites adding things at random from bits and piece they happen to find. It is really that bad, and every person coming along has to try to make sense of it. Shame on Google for not doing a socially responsible job. Shame on groups world wide who only care about their own incomes and selves first. Sorry, I am just tired. I spent twenty years just studying, then the last three years trying to get a few groups to change. Now I want to just label the videos with sky in them – to show all the data that is available — in context, right now, no lookups, no memorizing meaningless labels and words someone a thousand years ago used.
Richard Collins, Director, The Internet Foundation
D,
I did not know that it was an unfunded effort. I am a bit grumpy so I probably did not make clear that I don’t bother with groups who are not doing excellent work. I just know that there is a always room for improvement. Particularly with Internet practices.
If people have the source information – where and when an images was (is being) taken, that greatly improves the speed of the identification. Then the identification becomes a calibration and clarification of the image. Identification vastly improves the value of an image or sequence of images.
If your group is not training people to gather the information then you are not going to achieve your goals – unfunded or not. All the expertise and technical skills of some programs and algorithms does not make a global community. There are millions of outstanding technical groups in the world – and most of the fail to do more than inform a few.
FITS is maintained by a closed and small group. That makes it proprietary. Perhaps I should make a new term. “Single point of failure” comes to mind. “Too sharp and long learning curve”. FITS is one of the better formats on the Internet, but if you read the headers, they are often eclectic and hand made. There are no globally accessible FITS viewers, and most of the efforts using FITS treat it as though it were the end all of formats and do not also provide clear ways to use it with other groups. Yes, a few million people is laudable. But, on the Internet that is not enough.
A group can be closed for many reasons. The people do not identify themselves. The group assumes certain skills and tools,but does not assure that every new person can get them quickly and efficiently. The group is not global in its thinking and practices. Mostly only serves one country. I am pretty sure that the dialects of FITS in Europe are not compatible with the ones in the US. I don’t have time to gather a bunch of experiences to share with you on formats. I try to cover all of them evenly. And when any group begins to always say, “our format is open” but does not constantly find ways to remove barriers and improve – even to the point of abandoning the past – then, it is usually laboring under very difficult constraints and will grow much slower than other groups using different methods.
I think the unsupervised search is laudable. But why waste so much computer effort, when a starting estimate can vastly improve the speed. When I worked on the NASA GEM series of gravitational potential models, we used satellite orbits, and many sensors, to constrain the solutions. I was particularly good at taking new datasets and finding ways to get approximate solutions, then put them into the exact solver and run it out to a precise solution. A few minutes of a good estimator, can save months of computer time. I like that kind of problem, it seems especially elegant to go from years to days, or months to minutes. But there are lots of problems like that in the world. And the good groups, and good programs, and good solvers would be better used as part of larger efforts – each doing its part.
Your program ran, unsupervised, for a long time to solve an image. With a few extra pieces of information – that you could ask the users – it ran in a hundredth the time. Your pages can interact with the users – ask them what they are doing. You can find groups on the Internet who need image identification, registration, calibration – and don’t realize it. Find them and serve their needs – not wait for random chance to lead people to your site.
Go search for what you are doing – it is HARD to find your group. I already knew what your program was doing and how long it would take. What about the tens and hundreds of millions of people in the world who want to know the location of things in the sky that they see in their cell phones, in videos and images online.
It is 5 am. I got a couple of hours of sleep. I want your group to do better. If you have a global project, clearly identify yourselves and why you are doing what you are doing, know who needs your services – you can ask for funding. You might even get contracts and long term place in the world. I don’t have time or energy just now to look.
You still, after all these fairly direct and personal conversations, have not identified yourself. Told me who you are and why your are doing this. I don’t have time to look it up on the Internet, and what you think is there about your groups is not what most people will find if they search for you. I got good reviews from Eric about the value of the programs. But computers programs with undocumented embedded algorithms are NOT global communities.
“This project is partially supported by the US National Science Foundation, the US National Aeronautics and Space Administration, and the Canadian National Science and Engineering Research Council.” sound to me like you receive or have received funding at some point. I see you in the “people” page. “every useful astronomical image” is narrow. For every “astronomer” there are a thousand who want to know what is in the images they see. You have not listed the observatories, you have not found and maintain links to all the schools, all the groups, and individuals.
“legacy and badly archived” is changing daily on the Internet. Most of the astronomical data is being mined and reprocessed many times by many groups – none of them working together effectively on the Internet. They each do an OK job alone, but always fail to present a clear presence to the billions of Internet users.
Look at all the “professionals”. They are just ordinary people who might work full or part time or on their own. if they struggle and learn your particular tools and publish papers (most of them never read or used by any one), they are part of a very tiny group. Their hoarding or laziness in sharing with everyone is why the whole is so badly funded.
“We have elucidated and solved a fundamental computer science problem in the field of geometric hashing” — ABSOLUTELY. Nice, very nice. But others use that same algorithm and approach on different datasets, and they are just doing a good job. It is not a silver bullet. It is not the end of the universe. Your program has that speed so it can spend ten minutes of computer time looking unsupervised. But is that anything to crow about? “My computer can run through billions of triplets and quads of stars to search for a few stars in your image”. Yes, but if you don’t train the users and build communities and help them find those legacy and archived datasets – they will seem like they are doing a lot. “Our computer is running a google of comparisons” But not really helping people.
Your archived data is NOT the problem. A 1080P online video that is showing stars will have about 78 million frames a month. That is not getting used now. And the online video datasets are growing. Many of them are going to realize that sharing in video formats (lossless ones) is not a bad way to serve global audiences and markets. The observatories sneer at the private efforts, but the new private efforts are growing and people will get tired of large centralized expensive projects, paid with public moneys – supposedly for everyone – and only serving a few private investigators. I got really upset with LIGO and the Large Hadron Collider. They are throwing away most of the low energy but abundant, and earth based data that would serve the most people. Their decision makers only care about tiny problems and ignore the needs of the whole world. It is OK to have lofty distant goals, but if they do not serve the needs of society, we need better ways.
Now I really have to get going. Hope I have said a few things that might get you off your comfortable cushion and to look at what is actually going on with all the astronomy related sites on the Internet – I do that, and it appalls me. No two sites use the same methods of communicating, the waste of human time for billions of users, many of whom want to know more about the sky and stars and the solar system – are NOT being met. Not by a million fold. I count user time as fundamental. And society is wasting tens of years training people to memorize a few facts from a few books – when that could all be part of “let me look at these things together”.
I have looked at many hundreds of GitHub projects in detail over the past few years. It is extremely inefficient. The process relies on human memory in the loop for everything. It is fragile and breaks too easily. Mistakes made by individuals propagate too easily and are almost impossible to identify and repair. There is not much progress towards global collaboration on algorithm development – because ALL the groups, like yours, are working alone, unfunded and each making up their own tools and standards, or lack of them.
Where is the “donate button”?? And if you have NSF and NASA funding you are obligated (they are) to give clear links to the details. And probably are required to meet certain standards, and to share on places like Data.gov. There is no end of work to be done. But it takes a lot of work to find it and make it happen.
Send me a note when you get donate buttons on your GitHub and your site. There IS a way to ask for donations and support for GitHub and there is always Patreon, but I worry, because there is no oversight for Patreon. And no clear global values.
If your program gets used, that in nice. But if you build a global community that can be forever.
Richard Collins, Director, The Internet Foundation
