Hi, a fellow AF has found you can still search for AF threads via google using the "cache" link. Apparently google keeps copies of lots of stuff. For example he was successful in finding the wreckers list and the thread on STDT bosch conversions.

Not sure how to get beyond a few pages yet but will post when we work it out.

Could be useful.

Cheers

Jim

excellent news

if you can try and save as many of them as you can as it may well be useful

fingers crossed

I tink you'd find a lot more of the public domain posts than you might expect in the Google cache. Each new thread is assigned a number and that would have been a bit over 81500 by mid-October. There are then posts inside each thread and each is assigned a unique number. You can find these in the source file of the cached page.

If I use "aussiefrogs showthread.php?t=81514" as the Google search term, I get a Citroen Forum post from 5/10/10. Since the images in this thread were inline as opposed to added as thumbnails, it seems they are in the Google cache and can be recovered with the text. Changing the number gives another thread, but recent threads may be incomplete depending on when the page image was cached. Try a few random numbers and see what pops up. 81522 will let you relive someone else's recent trip, but the photos have been lost.

If it's not possible to recover the actual AF database records, then working through the cached posts by thread and post number seems to be one way to recover the information. It would need some structure and co-ordination to deal with a portion of the 80K+ threads, but saving the cached page as single file will preserve it. Extracting the individual posts in each thread from the page source would then be possible. You'd need to collect the cached pages by thread number, perhaps by assinging blocks to people to avoid duplication, and then have a script or code to extract the posts.

Just one idea / suggestion.

Perhaps unfortunately I think the toad pond will be uncached. However that could have benefits... So long as the actual car information can be saved we are part the way there.

Check out this link: boardreader - peujohn's 504 blog

I've been having a look around the web to see what I can find of my blog. This site has every post up until very recently, but the posts (except for the very short posts) are incomplete. Does anyone know if anything can be done with this site? If complete posts could be found it might be a big help in getting some info back.

Your blog is at thread number 65647 and you'll find the cached posts for much of it using the google search term:
aussiefrogs showthread.php?t=65647
All pages share the same thread number.

(Edit: Now found page 1 at:
http://cc.bingj.com/cache.aspx?q=aussiefrogs+thread.php%3ft%3d65647&d=4835501951352977&mkt=en-AU&setlang=en-AU&w=da2e512f,83b8b337
End Edit)

Page 2 is cached here (copy the whole link):
http://webcache.googleusercontent.com/search?q=cache:K58q2XIwOMgJ:www.aussiefrogs.com/forum/showthread.php%3Ft%3D65647%26page%3D2+site:aussiefrogs.com+aussiefrogs+showthread.php%3Ft%3D65647&cd=1&hl=en&ct=clnk&gl=au

(Edit: Page 3 exists via Bing (MS version of Google) and is here:
http://cc.bingj.com/cache.aspx?q=aussiefrogs+showthread.php%3ft%3d65647&d=4519070243294550&mkt=en-AU&setlang=en-AU&w=40502f52,686acd6b
End edit)

Some of the other pages seem to be cached at Google, but not all apparently. You have to look for the word 'cached' at the end of each entry and open it in a new tab. Save the whole page in whatever format suits, but the ".mht" option works. Page 2 includes a lot of pictures because they were originally posted inline.

It's also possible to find the cached summary pages for each forum by page. They are not necessarily up to date and the links obviously won't work, but you can at least extract the thread number of a topic of interest and then search for for it. Try the search term:
"http://www.aussiefrogs.com/forum/forumdisplay.php?f=11&order=desc&daysprune=-1&page=2"
This will find the cached page 2 of the Citroen Forum from August. 11=Citroen, 12=Peugeot, 13=Renault, 14=Chat,15-Cars, 18=Parts. The Pond (26) and Humour (16) fail to find anything, so they are probably not cached.

Peujohn, I've found and saved the cached version of all 11 pages of your blog. Pages 6 and 7 are not directly cached, but have been cached where one of the posts on the page was cached for some reason. Thumbnail images are missing, but there are lots of 504 pictures still there. pm me with an e-mail address and I'll send them to you. The files total around 30MB, so let me know if your what your mailbox will handle.

might have to get wget out to start spidering google!

OK, I've been pretty successful writing an aussiefrogs 'spider' script, (linux bash) which should give us (text) from 19th October which isn't too bad.

I'll try and recover and archive the renault forum in some sort of meaningful way as a gineapig, then hopefully let it loose on the site.

Saving the info is one thing, but then making meaning of it and forming it in a useful way will be challenging.

here goes!

**NB so far so good... combining (Google, Internet Archive, Bing, Yahoo)
------------------------------------------------------
Performed 11 queries against Live Search.
Adding 21115 URLs to URL queue for complete recovery.

David.

Web repo queries used:
ia 250 of 2000
google 9 of 1000
yahoo 42 of 5000
live 12 of 10000

There are 21199 URLs remaining in the URL queue.
Recovered 31 URLs so far.

Current time: Mon Oct 25 11:22:13 2010

Recovering [ 34] http://www.aussiefrogs.com/articles/605_review/index.htm ...

LOVE YA WORK

Top job vivid and David S. I'm planning to fully rebuild my blog, by reposting as much as I can, when Aussiefrogs is up and running again. I'd be happy to take on some other threads too. It's a lot of work but worth it.

well i have found caches going back to march 04 so there must be a hell of a lot of them floating around out there

i just did a search using my user name

Rambo...thats probably one of the better ways of doing it...we can all try and resurrect some of our "own" postings...also good time to filter out some of the crap (keep the valued and relevant stuff)...
Priority should be on preserving that that was left behind by people like AlanS and others that many on AF respect so much...

I think I understand the enormous task at hand and what has happend to AF is just dreadfull and unfortunate...but I think its important for us to take a positive attitude and a view to coming out of this with a new spirit...like phoenix from the flame...its just the right time of the year for spring clean...


cheers

dino

ps...Sometimes you dont know what you ve lost till its gone.SO
...another massive "thank you!" to all behind the scenes working hard to resurrect the mighty AUSSIEFROGS...

I don't know what the chances of getting the proper AF database back are, but there is a lot cached out there, other than the private forum sections. WayBack only has material up to about 2008, so it's use will be a bit limited as the probably about a third of the threads started later.

If cached material has to be used to recover posts, then probably the best place to start is the index pages. This is because it gives you the forum, the thread title, the thread number and the number of pages that need to be found. Then you can look for the threads on cached pages and extract the posts by number. I'll try to find some of the Citroen forum index pages and see how easy it is to make a list.

With the 504 blog, I needed to find the 11 cached pages from both Google and Bing, with two of them being found via the heading and page number rather than the thread number. It's not always immediately obvious.

Since the AF forum used a common board programme, there may even be an existing tool to extract the posts from a google cache and put them in a database. Anyone know??? Ideally, you'd like to be able to submit a cached page URL or web archive file as input, have a tool read it's source and record each post by thread and post number. If I had to, I know I could write something to do it given enough time, but someone needs to drive any attempt to capture and manipulate cached pages. That's obviously not me.

Thanks to David S, the almost complete 504 blog is now stored on my hard drive. A lot of the pics are included, some are not. The last post that has been recovered was October 10, Bob D's post about fitting 604 rotors to his 504. I'm not sure if this was the last post on the blog or not.

David, it would be good if you could put up some more details instructions for recovering the data. I'd like to have a shot at it.

John

Me too! I have lots of time, and love researching...but have no idea how to do techie stuff so instructions in laypersons terms would be greatly appreciated.

John, page 11 of your 504 blog was captured on 18/10/10, so not much would have been lost between 18 and 20/10.

The first thing you need to do is find the thread you want to recover. So John wants to find a thread on 504 power steering. Use the following search term at Google:
site:aussiefrogs.com aussiefrogs 504 power steering peujon
and you get a page full of stuff and you will scroll down and find the one with a link:
"www.aussiefrogs.com/forum/showthread.php?t=15417"
and 'cached' next to it. This tells us the thread number is 15417 and that a cached copy 'may' exist. It's page 1 because pages 2 and subsequent pages will display as such in the link description.

Try the search term "aussiefrogs showthread.php?t=15417" and you should find more than one page is cached for this thread. Page 1 of this thread doesn't seem to work, but open the cache of page 2 or 3. If it's what you want, use Menu> File> "Save As" to save a copy of the cached version. In Internet Explorer select .mht option and use a standardised naming convention so you can keep track of what you find. For example, use "AF_PF_15417_2_XX thread title etc XX.mht", meaning it's page 2 of AF thread 15417 and was in the Pug Forum. Easier to organise this way.

Since you've got a page 2, look at the top right of the original page and find out how many pages there were originally at the time the cache was captured (see the Google etc. header for the date). In this case there were 3 pages on 25/9/10. Save whatever pages you can find caches for and then look for the missing pages. There may be later caches too, but since this is an old thread, it probably doesn't matter.

Say, page 2 was missing but you had page 3. Take the page 3 link and change 3 to 2. So search on:
"www.aussiefrogs.com/forum/showthread.php?t=15417&page=3" OR
"aussiefrogs showthread.php?t=15417 page=3"
and you've got a fair chance of finding it.

Page 1's cached copy seems to be missing. There is a link, but it's not working. Try it later at:
http://webcache.googleusercontent.com/search?q=cache:-vWZxaOnp_wJ:www.aussiefrogs.com/forum/showthread.php%3Ft%3D15417+aussiefrogs+t%3D15417+%22504+-+power+steering%22&cd=2&hl=en&ct=clnk&gl=au

You can also try searching at Bing.com and other search engines that may have cached pages. Page 1 of this thread isn't there.

Does this help?

I think so

only one way to find out!

For the tech savy there is a couple of other options.

http://warrick.cs.odu.edu/warrick.html

http://www.guyrutenberg.com/2008/10/02/retrieving-googles-cache-for-a-whole-website/

Just watch how often you hit google etc .. or they will ban your IP. Maybe a couple of dozen people do it and combine results?

Also go to google images http://www.google.com/imghp?hl=en&tab=wi and search for
site: aussiefrogs.com

I just know this will show my ignorance in all things IT related, but is it not possible to approach google and ask them to do it?

Mike.

So far I've saved Jo Proffi's member gallery, a thread on heater cables and another on auto chokes in fuegos, also mistarenno's thread on fuego replacement wheels and a couple of other bit's & bobs

are board reader pages useful or not? They have the text but the format is different.

that script might work alright. If you search for aussiefrogs on google you get 10500 pages though. Might take a while! I've recovered most of my 504 turbo blog, although page 2 dosn't have a cache on google.

I think you could get around the banning from google by adding sleep timers to the script so it dosn't just smash google with 10500 requests. Make it sleep for a bit. Will increase the time of execution sure, but if you just leave it to go then it should be fine.

I've set up warrick on my server at home and will just keep it running and see what it pulls out. I tried to do it on a thread by thread basis, but with the structure of the urls, I might as well do the whole lot. I'm doing all of the archives with the exception of Google to avoid over-use problems.

I'll let you know how it goes.

Matt

Web repo queries used:
ia 1999 of 2000
google 11 of 1000
yahoo 1209 of 5000
live 1253 of 10000

There are 28031 URLs remaining in the URL queue.
Recovered 1215 URLs so far.

Current time: Tue Oct 26 14:13:18 2010

The ia repository has run out of daily queries.

Sleeping 24 hours...
Hour 1
Hour 2
Hour 3
Hour 4
Hour 5
Hour 6
Hour 7
Hour 8
Hour 9
Hour 10
Hour 11
Hour 12

Web repo queries used:
ia 1318 of 2000
google 4 of 1000
yahoo 858 of 5000
live 882 of 10000

There are 32717 URLs remaining in the URL queue.
Recovered 2104 URLs so far.

Current time: Wed Oct 27 22:11:44 2010
.
.
.
It's a loooong road, hows everyone else going?

David.

Vivid

I'm going well but around here the only one that can read chinese is the wife. The only thing that makes sense in all of the above is
Current time: Wed Oct 27 22:11:44 2010
.
.
.
It's a loooong road, hows everyone else going?

David.

Thanks to you and those who do know what they are doing - hopefully there are 16 who are doing/have done the same as you. At the risk of sounding really dumb, not an uncommon thing with me, how do you know that someone else is not saving the same as what you have already saved?
If it is a simple process I'm more than ready to help, but i will need some very basic instructions

Mine's still trundling along with warrick for the last couple of days. I haven't had a chance to have a good look at what's there, but I'll go through it on the weekend if I get a chance.

As my server remains on all of the time I'll just let it run. I haven't bothered looking for any threads in particular. Have just done a recursive search from the main site. I did notice though that it's pulling a lot of cached pages of aus-city now, so i'm not sure if that will affect things.

Matt

If anyone can find me any threads on repacking the aux belt tensioner on the XU engines. Posts or threads by AlanS usually.
My comp tech skills suck so I'd not know where to start.
Thanks

Google seems to have the most cached material, followed by Bing and Yahoo. I think WayBack/IA will only be useful for some older threads.

It seems the last thread and post numbers were probably around 82,000 and 831,000, respectively. There's obviously a good portion of this that will never be recovered and a lot of those posts would have been either in threads that no longer matter or in 'private' threads that would seem to be simply unrecoverable as they were never cached. I haven't found anything at all from the Toad Pond, so I assume it's lost and Ken will have to restart that GW/AGW thread.

I've manually captured a good portion of the public thread archive index pages. Each page relates to a specific forum and has up to 250 thread titles listed. These will be in the order they were created as the overall site thread count would no doubt increment by 1 whenever anyone creates a new thread somewhere. For example, there are 73 pages of 250 for the Pug Forum, being almost 18,000 threads. The thread numbers can be extracted from the links in the source, but I've only been grabbing the cached pages so far. If the thread number is known, then it's easier to search for and find specific cached pages that are part of that thread. Numerous threads will have the same title! The automated cache grabbing is necessary to do it quickly, but some threads should be searched for and saved individually.

So far I've been able to find archive index pages for Social (Forum 10 Page 4 only of 5), Citroen (F11 29/35 pages), Peugeot (F12 54/72), Renault (F13 18/30), Simca (1/1), Chat (F14 13/26), Cars (F15 13/22), Tech (F17 7/19), Parts (F18 27/37), Chat Archive (F19 1/1), Motorsports (F24 3/5), Simca (F28, 1/1). The Simca forum has only one page with 228 entries from the start. Some were easy to find, other not so obvious. A lot of the links found went nowhere as they were not for cached pages.

Do we want to hand out pages of 250 threads to selectively search for and save? Not all will be found or even worth finding? Some should probably be ignored now, like the 'spotted' threads, but cars and parts threads can be useful.

Any thoughts?

Send some of the Peugeot pages to me David, I'll go through them and see what I can save. You have my email address. I'll be away for the next few days though.

For the sake of others, this is pretty easy to do. The archive pages have a list of threads. You can right-click on each link to a thread and select "Copy link address" then paste this into Google etc as a search term. If Google finds the page (it won't always), you should click to repeat the search with the omitted results included. This will give a list of links to each page of the thread (if there is more than one page). Then you can open each page by clicking the "Cached" link, then choose Save from the File menu in your browser.

Is that the way you do it David?

I suppose the next question is who do we send the cached pages we've saved to? (although my contribution to the recovery is puny and likely only doubles up on the threads found by others)

David, If it's not too much trouble, could you please send them to me ?
chadi.sawires@gmail.com