Project 1999

Project 1999 (/forums/index.php)
-   Green Server Chat (/forums/forumdisplay.php?f=73)
-   -   Searchable portal for the Everquest Archives (/forums/showthread.php?t=373614)

Dolalin 12-08-2020 04:16 PM

Searchable portal for the Everquest Archives
 
4 Attachment(s)
Some may be aware that I do a lot of research.

Over the course of the past two years I've collected over 4 million files while dumping Everquest-related websites, deprecated Yahoo Groups mailing lists, and more.

Making these easily searchable has been a constant puzzle. But I've now been able to leverage Azure Cognitive Services and Azure Cognitive Search to create a searchable portal to the eq archives.

Presenting, the Everquest Archives search portal:

https://eqarchives.azurewebsites.net/

So far I have been able to index 335,000 of the 4 million files, including all the Yahoo Groups mailing lists. They are context-searchable and metadata fields will give you links to the original Wayback Machine archive link.

Indexing the archives is proving expensive, so I will be soliciting donations at a future time to continue the process. For now, you can enjoy the 335,000 indexed documents, which include the Sony boards, EQ Stratics, Casters Realm, and a few others (alphabetically, the contents of /websites/ in the Git archive up to the letter E).

Let me know what you think!

loramin 12-08-2020 04:23 PM

https://i.imgur.com/ZeLI8p3.gif

Danth 12-08-2020 04:28 PM

Funny enough I tried to PM you a couple weeks ago as to whether you could search for some specific things, except I couldn't because you seem to have forum PMs turned off. What a nice looking resource you've built. There are a few items I intend to search for as time and motivation permit.

Danth

Dolalin 12-08-2020 04:35 PM

I noticed there's a bug on mobile where if you search for something and press the enter button instead of the magnifying glass, your result gets reset to * on the results page. Really annoying, I will try to fix it when able.

For now, if you're on mobile, press the magnifying glass to start your search.

mcoy 12-08-2020 04:35 PM

This is awesome! Any chance you have Graffe's on your list of sites to index/archive?

-Mcoy

Dolalin 12-08-2020 04:38 PM

Quote:

Originally Posted by mcoy (Post 3225002)
This is awesome! Any chance you have Graffe's on your list of sites to index/archive?

-Mcoy

It's in the Git archive but I haven't gotten that far yet with indexing (G is after E, hehe).

It is quite expensive to index this. I need to firm up the pricing from Azure then I will ask for donations to continue the process.

Izmael 12-08-2020 04:43 PM

Great initiative.

I can provide mirroring of this if needed, just let me know. Never know what will happen to the main site in a few years.

Dolalin 12-08-2020 04:55 PM

1 Attachment(s)
The hosting is pretty easy, the site is just a tiny .netcoreapp, but it's backed by an Azure Search Service which is.... harder to mirror. I'm actually not sure if the data can be exported. :confused:

Dildy 12-08-2020 04:59 PM

HNGHHHH great work! Can't wait to dive in.

Artelius

Jibartik 12-08-2020 05:08 PM

Quote:

Originally Posted by loramin (Post 3224990)



All times are GMT -4. The time now is 02:01 PM.

Powered by vBulletin®
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.