Page 1 of 2
user-tailored "most wanted list"
Posted: Mon Nov 17, 2008 3:49 pm
by uigrad
When people use the most wanted list, it's good for the site (because it fills in the holes), and it's good for the user's stats (since they'll find a lot of users that need updates). But, to use the most wanted list, I generally have to delete characters I've made on other realms, create a new character for the realm, and then run the census.
I have low level characters on a bunch of realms, and don't want to make any more, so I prefer to check how many updates (in the last 30 days) exist for my realms, and use that to decide which ones need a census.
I built a simple perl script that will query the realm data pages at WR, and show me the number of updates over the last 30 days for each of my realms:
Code: Select all
#!/usr/bin/perl
@serverlist = qw(741 783 80 42 85 654 32 77 21 26 46 67 43 535 57 49 559);
foreach $id (@serverlist) {
$document = `lynx -nolist -pseudo_inlines -dump "http://www.warcraftrealms.com/census.php?serverid=$id"`;
@doc = split("\n", $document);
$server = IndexOf("Server: ", \@doc);
$ratio = IndexOf("Ratio: ", \@doc);
printf ("%4sA,%4sH %s\n", $doc[$ratio+2] =~ /\s+(\d+) : (\d+)\*/, $doc[$server] =~ /(.*) GO/);
}
sub IndexOf { # pass in value, array reference
my ( $value, $arrayref ) = ( shift, shift );
foreach my $i ( 0 .. @$arrayref-1 ) {
return $i if ($$arrayref[$i] =~ /$value/);
}
}
Output looks like:
Code: Select all
14A, 7H Server: [Antonidas................]
92A, 5H Server: [Arathor..................]
31A, 61H Server: [Burning Blade............]
2A, 133H Server: [Crushridge...............]
17A, 6H Server: [Durotan..................]
23A, 5H Server: [Executus.................]
47A, 68H Server: [Feathermoon..............]
20A, 45H Server: [Hellscream...............]
6A, 24H Server: [Illidan..................]
40A, 111H Server: [Mal'Ganis................]
258A, 355H Server: [Medivh...................]
59A, 36H Server: [Shattered Hand...........]
160A, 312H Server: [Silvermoon...............]
16A, 17H Server: [Smolderthorn.............]
89A, 45H Server: [Stormrage................]
42A, 62H Server: [Stormscale...............]
166A, 65H Server: [Twisting Nether..........]
Based on that output, I can see that Crushridge has had plenty of horde updates (133) recently, but only 2 alliance updates. It would be a prime place for me to run a census.
Since the warcraftrealms.com site knows the realms that I have characters for, it would be nice if it could print such a customized list for me.
Rollie, I know you're unlikely to implement this because of all the other things on your wishlist, and that's fine. I'm hoping someone else finds my script useful, though. It should run fine from any command line in unix. Windows users might be able to run it using cygwin.
Posted: Thu Nov 20, 2008 10:27 pm
by SannaSK
you are brilliant. i was just wondering if there were some way i could do this. no way could i have built this code on my own. now i just have to find some way to run this. can i put it in a ... webpage somehow? i have WinXP64, i'd like to avoid installing an entire program just for this.
When I started making my armyhorde of alts (all named Jaecen, as my main is Jae and I'm taking census : p), I realized real quick I was going to lose track of who-what-when-where-why-omfg, so I bookmarked each server in a folder in Firefox, and just thought, "well, I can open them all in tabs." And Firefox went "lolnoob you're opening 26 tabs, are you sure you want to do this?" and i kind of went
.
so you get massive internet cookies for this. yay! now i just have to do something with it.... *ponders*
Posted: Fri Nov 21, 2008 9:38 am
by Rollie
Slick script =)
Noobcoder's Misadventures with Perl.
Posted: Fri Nov 21, 2008 3:06 pm
by SannaSK
Noobcoder's Misadventures with Perl.
So first I went to my website and made a wr_cen.pl file in the base directory, into which I C+Ped the entirety of the script there, no alterations. When I attempted to load the page, I got a 500 Internal Server error. Asked a couple of friends, moved the file to the /cgi-bin/ folder, and got a blank white page.
Headkeyboarded for a bit, then broke down and installed Cygwin, as well as another program called Strawberry Perl. Opened them, attempted to copy-paste the script directly into the black-background box, and all that came up was a ^V. My friend said 'roflnoob, you have to put the code into a .pl file on your computer.' I said, 'oh, well that makes sense at least.'
So I've got this wr_cen.pl and durned if I can figure out how to make the little command line box thinger access it. I am such a perl noob. how i learned html, php, mysql is beyond me --owait, i am epixphail at php/mysql, that probably answers that-- but yeah, to anyone who doesn't already know what to do with a perl script, don't try this at home yet. When I figure out what to do (which really means 'when my more knowledgeable friends tell me what i'm doin rong), I'll edit this post with Direcshuns For Perl Dummies.
Until then, I guess I just, uh... open my 26 tabs... : <.
edit 1, friday morning. my friend says I need to install Lynx, a standalone text-only browser. Do you suppose if I alter that to be 'firefox,' would it still work? Maybe that's why my attempts at running it on my webhost didn't work, it was looking for this lynx thing that didn't exist : p
edit 2, friday midnight. Ok, my friend figured out that when I installed Cygwin, all the alternate packages that I thought were set to install were specifically set to /not/ install. Fixed that by rerunning the setup and installing perl (4 reelz this time) and lynx. currently stuck at the 2 "bad interpreter" errors. still working.
edit 3, sat noonish. the internet makes me think there's something with Windows CFLRs in my notepad file that's screwing up the command line's ability to process mah file. Just to be clear, my friend can run the script fine on her linux box; i'm not trying to say my confusion is coming because there's something wrong with the script. it's totally not, the script is fine. everyone else can run it... just not me, because i'm an effing noob : p.
edit 4, sat noonish but about an hour later than the above. progress! typing 'perl c:/tempy/wr_cen.pl' stopped giving me error messages and just did nothing. (that's pretty sad when that qualifies as progress.) i searched for 'lynx' in the cygwin folder and got nothing that remotely looks like an EXE file, which i find strange but don't know what to do about. in the $document line of the .pl file, i replaced 'lynx' with 'firefox' and ran the command again. got repeating lines of
"sh: firefox: command not found
A, H"
so it's halfway working. I don't know what I did that it was giving me the error messages and then it wasn't.
Have primarily come to the conclusion that this is not a recommendable way for your average Warcraft Realms census taker to create their own customizable most wanted list. : p. However, still working on this.
edit 5, about 2 minutes later than above. First I'm going to multiclass to rogue just long enough to stab Cygwin in the faise, then truss it up by its toenails and stab it some more in other soft parts of its body. It specifically /did not install/ the Lynx.exe and related folders, because i'm missing some other .dll somewhere. ARGH. still working.
EDIT 6, about 15 minutes after above: SUCCESS, so far as I can tell. I can't open the lynx.exe directly, it QQs about can't find a 'cygcrypto-0.9.8.dll' file (which exists just fine in the cygwin/bin directory). i reinstalled the packages i had chosen before, chose a different mirror, and all of a sudden it WORKS now. the Cygwin command line gives me the nice happy text display like it does above. Mwahahaha!
Damned if I know what I did tho, esp in regards to the suddenly-disappearing 'bad interpreter' returns. But it works now! Yay!!
Next on the list: figure out why UniUploader hates me. but that's for another thread : [.
edit 7, about 30 minutes later. I lied. the command line box queries two servers and locks up. *sigh* i give up. in this amount of time, i could probably have coded a php script to do everything server-side and just handed it to rollie. well maybe i could have. i would have been far less clueless about this whole mess. *tears some hair out*
Posted: Sat Nov 22, 2008 8:56 am
by Rollie
So I thought I'd give it a shot myself.
Copy/Pasted the above code into test.pl on my server. From the command line (I use putty), ran
perl test.pl
And it worked like a charm!
Do you have shell access to your web server?
Posted: Sat Nov 22, 2008 10:35 am
by SannaSK
Shell... my webhost runs CPanel, and while there is an option that says "SSH/Shell Access," clicking it only gives me an option to set up an SSH key, which I did in the hopes it would then give me /another/ option which it did not. Which is odd, because the video that it shows clearly displays the test-user having the ability to open a shell window from that page. Maybe I have to go enable something somewhere else.
TLDR i do 'have' shell if only i can figure it out.
Edit.
ok. What ended up happening is this.
After myself and my wonderful friend J spent 2 days bashing our heads against the Cygwin install on my machine, my other friend M shows up and tells me to put the line
print "Content-type:text/html\n\n";
on the line before
foreach $id (@serverlist) {
and SOMEHOW. SUDDENLY. the /cgi-bin/wr_cen.pl file on my webserver works. Not the one on my machine that I tried to use with Cygwin, but the one on my webserver randomly works now. I'm one enterbreak short, because I get the results as a long single line of text, but that works for my personal preference anyway. I C+P all of it into a notepad anyway so I can delete the server's line when I've taken census on that server today.
This is not a viable solution for personalized most wanted lists for individuals who aren't already very familiar with things like 'perl, scripts, cgi-bin.' If you have to install anything on your computer that you're feeling iffy about, this is not for you (unless you have a really good friend willing to hold your hand through like every fraggin step, and then some). On the other hand, once it works, it works like a charm (assuming your Cygwin doesn't freeze up every single time).
Thanks very much, OP. b^_^b Army of census alts, release the hounds!
Posted: Tue Nov 25, 2008 5:47 pm
by uigrad
I'll put out a shameless plug for freeshell.org. They let you open a shell account for a dollar (used to be free, but they had problem with scammers).
Step 1) download putty (best freeware terminal for windows)
Step 2) use PuTTY to telnet to freeshell.org, use 'new' as account name.
Step 3) send a dollar bill to the address it gives you to finish account creation
Step 4) run any perl scripts or CGI scripts that you want.
Alternatively, you could use silenceisdefeat.org. I think it is similar to freeshell.org, and it lets you start an account with a dollar, and allows you to send it through paypal. I haven't used it as much.
edit: by the way, your friend is correct about the Content-type. If you serve static documents, then the webserver uses a combination of things (file extension, local .htaccess file, global settings) to figure out how to serve it. If the document is generated by a CGI script of any kind, then you need to provide the content header yourself.
Alternatively, you could put this at the top of your document:
Code: Select all
use CGI qw/:standard/;
print header();
Posted: Thu Apr 01, 2010 4:08 pm
by SannaSK
Pardon my necro on this thread X D
when editing a .pl, does the encoding type matter? After much stabbiting and a great deal of assistance from more knowledgeable friends, I finally got this script to run on my website. After a long absence away from CensusPlus, I picked it back up again and went into my website to try to update the serverlist with the servers I had chars on.
And now the .pl won't load at all. It gives me a giant string of
A, H
A, H
A, H
and no actual data. I'm rather distressed, as my only other option is the unwieldy method of bookmarking each server in warcraftrealms.com and opening them all in tabs, which seems like an unkind thing to do to Rollie's bandwidth.
/all/ I did was open the .pl, paste in the new string of numbers, and save. The only thing I can think is that the encoding type matters. It's currently set to us-ascii. Is this good? Bad? Did it break for some completely unrelated reason?
Posted: Fri May 21, 2010 7:38 am
by Hybuir
If someone has a working site with this script, this may just bring me back to the board lol
Posted: Sun May 23, 2010 7:35 pm
by Kosh
@SannaSK, since there aren't any non-ASCII characters in the script, the character encoding shouldn't matter, as long as it's some form of "Western" encoding. It's more likely that you somehow broke the script with your alteration, or broke the line-endings by using a bad editor (I believe Notepad does this). UNIX uses line-feed (^J), and MS-DOS's carriage-return / line-feed combination tends to break a lot of UNIX tools.
If you still have your original version of the script, try running it, instead. If you still get the blank "A, H" lines, that would indicate that lynx isn't working for you anymore (or can't reach the website).
Now, thanks to Hybur for bumping this, I decided to try this script out for myself. I like what it does, but realized that I am also interested in the "Update dates". Here's my modified script to add the two dates at the end of each line:
Code: Select all
#!/usr/bin/perl
@serverlist = qw(741 783 80 42 85 654 32 77 21 26 46 67 43 535 57 49 559);
foreach $id (@serverlist) {
$document = `lynx -nolist -pseudo_inlines -dump "http://www.warcraftrealms.com/census.php?serverid=$id"`;
# $document = `cat tmpcen.txt`;
@doc = split("\n", $document);
$server = IndexOf("Server: ", \@doc);
$ratio = IndexOf("Ratio: ", \@doc);
$update = IndexOf("Update ~ ", \@doc);
chop( $serverline = `echo "$doc[$server]" | sed -e 's/ */ /g;s/\] GO.*/]/;y/./ /'` );
printf( "%4sA,%4sH %s (%s,%s)\n", $doc[$ratio+2] =~ /\s+(\d+) : (\d+)\*/, $serverline, $doc[$update] =~ /.* ~ (.*)/, $doc[$update+1] =~ /.* ~ (.*)/ );
}
sub IndexOf { # pass in value, array reference
my ( $value, $arrayref ) = ( shift, shift );
foreach my $i ( 0 .. @$arrayref-1 ) {
return $i if ($$arrayref[$i] =~ /$value/);
}
}
And here's the output from that serverlist:
Code: Select all
328A, 375H Server: [Antonidas ] (2010-5-22,2010-5-22)
141A, 4H Server: [Arathor ] (2010-5-23,2010-5-18)
13A, 10H Server: [Burning Blade ] (2010-5-23,2010-5-23)
3A, 217H Server: [Crushridge ] (2010-5-21,2010-5-21)
27A, 177H Server: [Durotan ] (2010-5-22,2010-5-21)
147A, 4H Server: [Executus ] (2010-5-19,2010-5-20)
29A, 190H Server: [Feathermoon ] (2010-5-23,2010-5-23)
170A, 489H Server: [Hellscream ] (2010-5-22,2010-5-22)
127A, 118H Server: [Illidan ] (2010-5-20,2010-5-20)
52A, 184H Server: [Mal'Ganis ] (2010-5-22,2010-5-22)
28A, 7H Server: [Medivh ] (2010-5-17,2010-5-15)
325A, 103H Server: [Shattered Hand ] (2010-5-23,2010-5-23)
29A, 225H Server: [Silvermoon ] (2010-5-23,2010-5-23)
5A, 317H Server: [Smolderthorn ] (2010-5-10,2010-5-18)
138A, 48H Server: [Stormrage ] (2010-5-22,2010-5-22)
2A, 3H Server: [Stormscale ] (2010-5-19,2010-5-18)
44A, 161H Server: [Twisting Nether ] (2010-5-22,2010-5-22)
I also didn't really like all of the extra space around the Server (and the dots after the name), so I took the opportunity to "fix" those, as well. As a bonus, this keeps each line under 80 characters, so it will still fit on even a narrow (80-column) terminal. Note that I used "sed" to accomplish this, so if your installation somehow has Perl and lynx, but
not sed (!?), this version won't work for you.
If you liked the dots, but otherwise like this version, you can just remove the ";y/./ /" from near the end of like 13.
Posted: Tue May 25, 2010 8:32 am
by Hybuir
moar dots!
(I kid)
Posted: Tue May 25, 2010 11:09 pm
by Kosh
Hybuir wrote:moar dots!
(I kid)
Oh don't worry, I can whip up an extra-special, "moar dots" edition, just for you
Posted: Thu May 27, 2010 6:29 pm
by Kosh
I was thinking about all of those Windows people who can't run scripts, and since it will probably be even longer than Blizzard's definition of Soon? before Rollie implements something like this on the site, I decided to whip up my own web-service.
Now, I'm not a GUI person, so it isn't "pretty", and it's on a personal machine, so if it starts getting hammered, I'll just pull the script, but it can be found at
http://mackosh.dyndns.org/~kosh/censerv/censerv.php.
How it works is, you pass the list of serverIDs you are interested in to the script by specifying "?servList=" and your list of IDs in the URL. You can use commas or spaces between the numbers, but since spaces are illegal in a URL, your browser
should replace them with "%20", making the list longer and harder to read.
I make
no effort to manage cookies, so you will always need to specify the "servList", but you can bookmark it once you have the list the way you want.
To pick on the OP's server list, here's what the URL would look like:
(big long url with long list of numbers -- hover or edit post to see it (or click the link))
I implemented a cache, so if multiple people ask for the same ID (or you keep hitting "refresh"), Rollie's server won't keep getting hit. I've currently got the cache time set at an hour; how often does the data behind those pages get updated, Rollie?
I haven't gotten around to implementing "dots" / "moardots" mode, but I'll mention one of the options that is there now: "verbose". The page defaults to verbose, but you can make it
slightly less so by appending "&verbose=false" to the URL.
[Edit]prettify second link so it doens't force the page width out[/Edit]
Posted: Fri May 28, 2010 12:54 am
by Kosh
Well, I've already got a hit from GoogleBot, which encouraged me to get robots.txt set up for that domain (I have metadata in the page not to cache, etc. it, but of course the page is already loaded by then).
To make sure that other spiders (whether well-behaved or not) won't kick off queries when following the second link above, I've added another option to my script, and edited that link to add "&cache=only". This will return the cached information, even if it has expired.
You (probably) don't want to enable that option for your normal use.
Posted: Fri May 28, 2010 2:10 pm
by 1974ER
Thank you Kosh, I bookmarked the page with my little personal list.
Posted: Fri May 28, 2010 11:18 pm
by Kosh
You're welcome
Being an "Epic Censi", is it really a "little list", though?
I'm glad it's useful to some people. As of when I write this, 29 different server IDs have been viewed (including 1, 2, 3, 4, the example list I give you with no list
), so we'll see if more people notice it / start using it.
Oh, and I masked the second URL in my prior post, so I didn't have to side-scroll anymore.
Posted: Sat May 29, 2010 1:37 am
by 1974ER
Hehehe
:D:D , at least for the time being, yes. I just wanted to test it and put in just four servers on which I actually play on, not to overload your computer or WCR.
Edit: I will probably increase it to about 20 later on, to cover additionally those realms where I have "permanent" census alts. As for being "Epic Censi", well, Rollie deemed it appropriate to surprise me with the title a while back. As for reaching the figures I have... well, I have gone through quite a large number of deleted level 1 alts. Having only one account and an old computer to work with places its limits on censusing efforts.
All and all, I am pretty happy to note that I might break the 10M updates barrier near the end of June / early July, depending on how big a hit trying to do lots of Midsummer Fire Festival quests will take on my submission figures during that time period.
Posted: Sat May 29, 2010 1:05 pm
by Kosh
Don't worry. I query each page from WR sequentially, so more numbers just take longer. The bigger load would be if multiple people are all querying different (uncached) numbers at the same time.
I don't currently impose any limit to the number of serverIDs you can request; the only limits are system limits on URL / variable length. I know that 17 IDs work just fine, since that's what is in the example URL.
It's looking like Rollie's process to update the server-level stats runs sequentially across the servers, since I have noticed that different servers get refreshed at different times. That probably throws out any kind of "smart" caching (when the cache lasts longer when no updates are expected), but I might bump up the cache life to two or three hours.
Good luck with your "army".
Oh, and we're up to 34 unique IDs now (and someone discovering that "All" isn't a valid ID :p ).
Posted: Sat May 29, 2010 5:27 pm
by FuxieDK
and someone discovering that "All" isn't a valid ID :p
That would be me
Posted: Sat May 29, 2010 6:47 pm
by Kosh
FuxieDK wrote:and someone discovering that "All" isn't a valid ID :p
That would be me
Seriously, even if I were nutty enough to implement "All", would you really want such a page?
Don't worry, it's a good test of my input validation.
Speaking of validation, the generated pages
do validate, amazingly enough.
Oh, and Hybuir, just for you, I've added a "very special" feature... Just add "&moardots" to a query.
