Stripping HTML

[helios]

I'm writing a simple web crawler, and I need to create files based on information I get from an HTML file. The problem is that I don't want to create files named "foo". "foo" is what I'm looking for.
Does anyone know of a reference with a simple HTML character entities to UCS conversion table that can be easily processed by a computer? Ideally, an entry should look something like ""\t0022".

EDIT: Damn. Every time I ask a question, I end up finding the answer five second later.
Here's a processed table, in case anyone needs it:

std::map<std::string,wchar_t> entities;
entities["quot"]=(wchar_t)0x0022;
entities["amp"]=(wchar_t)0x0026;
entities["apos"]=(wchar_t)0x0027;
entities["lt"]=(wchar_t)0x003C;
entities["gt"]=(wchar_t)0x003E;
entities["nbsp"]=(wchar_t)0x00A0;
entities["iexcl"]=(wchar_t)0x00A1;
entities["cent"]=(wchar_t)0x00A2;
entities["pound"]=(wchar_t)0x00A3;
entities["curren"]=(wchar_t)0x00A4;
entities["yen"]=(wchar_t)0x00A5;
entities["brvbar"]=(wchar_t)0x00A6;
entities["sect"]=(wchar_t)0x00A7;
entities["uml"]=(wchar_t)0x00A8;
entities["copy"]=(wchar_t)0x00A9;
entities["ordf"]=(wchar_t)0x00AA;
entities["laquo"]=(wchar_t)0x00AB;
entities["not"]=(wchar_t)0x00AC;
entities["shy"]=(wchar_t)0x00AD;
entities["reg"]=(wchar_t)0x00AE;
entities["macr"]=(wchar_t)0x00AF;
entities["deg"]=(wchar_t)0x00B0;
entities["plusmn"]=(wchar_t)0x00B1;
entities["sup2"]=(wchar_t)0x00B2;
entities["sup3"]=(wchar_t)0x00B3;
entities["acute"]=(wchar_t)0x00B4;
entities["micro"]=(wchar_t)0x00B5;
entities["para"]=(wchar_t)0x00B6;
entities["middot"]=(wchar_t)0x00B7;
entities["cedil"]=(wchar_t)0x00B8;
entities["sup1"]=(wchar_t)0x00B9;
entities["ordm"]=(wchar_t)0x00BA;
entities["raquo"]=(wchar_t)0x00BB;
entities["frac14"]=(wchar_t)0x00BC;
entities["frac12"]=(wchar_t)0x00BD;
entities["frac34"]=(wchar_t)0x00BE;
entities["iquest"]=(wchar_t)0x00BF;
entities["Agrave"]=(wchar_t)0x00C0;
entities["Aacute"]=(wchar_t)0x00C1;
entities["Acirc"]=(wchar_t)0x00C2;
entities["Atilde"]=(wchar_t)0x00C3;
entities["Auml"]=(wchar_t)0x00C4;
entities["Aring"]=(wchar_t)0x00C5;
entities["AElig"]=(wchar_t)0x00C6;
entities["Ccedil"]=(wchar_t)0x00C7;
entities["Egrave"]=(wchar_t)0x00C8;
entities["Eacute"]=(wchar_t)0x00C9;
entities["Ecirc"]=(wchar_t)0x00CA;
entities["Euml"]=(wchar_t)0x00CB;
entities["Igrave"]=(wchar_t)0x00CC;
entities["Iacute"]=(wchar_t)0x00CD;
entities["Icirc"]=(wchar_t)0x00CE;
entities["Iuml"]=(wchar_t)0x00CF;
entities["ETH"]=(wchar_t)0x00D0;
entities["Ntilde"]=(wchar_t)0x00D1;
entities["Ograve"]=(wchar_t)0x00D2;
entities["Oacute"]=(wchar_t)0x00D3;
entities["Ocirc"]=(wchar_t)0x00D4;
entities["Otilde"]=(wchar_t)0x00D5;
entities["Ouml"]=(wchar_t)0x00D6;
entities["times"]=(wchar_t)0x00D7;
entities["Oslash"]=(wchar_t)0x00D8;
entities["Ugrave"]=(wchar_t)0x00D9;
entities["Uacute"]=(wchar_t)0x00DA;
entities["Ucirc"]=(wchar_t)0x00DB;
entities["Uuml"]=(wchar_t)0x00DC;
entities["Yacute"]=(wchar_t)0x00DD;
entities["THORN"]=(wchar_t)0x00DE;
entities["szlig"]=(wchar_t)0x00DF;
entities["agrave"]=(wchar_t)0x00E0;
entities["aacute"]=(wchar_t)0x00E1;
entities["acirc"]=(wchar_t)0x00E2;
entities["atilde"]=(wchar_t)0x00E3;
entities["auml"]=(wchar_t)0x00E4;
entities["aring"]=(wchar_t)0x00E5;
entities["aelig"]=(wchar_t)0x00E6;
entities["ccedil"]=(wchar_t)0x00E7;
entities["egrave"]=(wchar_t)0x00E8;
entities["eacute"]=(wchar_t)0x00E9;
entities["ecirc"]=(wchar_t)0x00EA;
entities["euml"]=(wchar_t)0x00EB;
entities["igrave"]=(wchar_t)0x00EC;
entities["iacute"]=(wchar_t)0x00ED;
entities["icirc"]=(wchar_t)0x00EE;
entities["iuml"]=(wchar_t)0x00EF;
entities["eth"]=(wchar_t)0x00F0;
entities["ntilde"]=(wchar_t)0x00F1;
entities["ograve"]=(wchar_t)0x00F2;
entities["oacute"]=(wchar_t)0x00F3;
entities["ocirc"]=(wchar_t)0x00F4;
entities["otilde"]=(wchar_t)0x00F5;
entities["ouml"]=(wchar_t)0x00F6;
entities["divide"]=(wchar_t)0x00F7;
entities["oslash"]=(wchar_t)0x00F8;
entities["ugrave"]=(wchar_t)0x00F9;
entities["uacute"]=(wchar_t)0x00FA;
entities["ucirc"]=(wchar_t)0x00FB;
entities["uuml"]=(wchar_t)0x00FC;
entities["yacute"]=(wchar_t)0x00FD;
entities["thorn"]=(wchar_t)0x00FE;
entities["yuml"]=(wchar_t)0x00FF;
entities["OElig"]=(wchar_t)0x0152;
entities["oelig"]=(wchar_t)0x0153;
entities["Scaron"]=(wchar_t)0x0160;
entities["scaron"]=(wchar_t)0x0161;
entities["Yuml"]=(wchar_t)0x0178;
entities["fnof"]=(wchar_t)0x0192;
entities["circ"]=(wchar_t)0x02C6;
entities["tilde"]=(wchar_t)0x02DC;
entities["Alpha"]=(wchar_t)0x0391;
entities["Beta"]=(wchar_t)0x0392;
entities["Gamma"]=(wchar_t)0x0393;
entities["Delta"]=(wchar_t)0x0394;
entities["Epsilon"]=(wchar_t)0x0395;
entities["Zeta"]=(wchar_t)0x0396;
entities["Eta"]=(wchar_t)0x0397;
entities["Theta"]=(wchar_t)0x0398;
entities["Iota"]=(wchar_t)0x0399;
entities["Kappa"]=(wchar_t)0x039A;
entities["Lambda"]=(wchar_t)0x039B;
entities["Mu"]=(wchar_t)0x039C;
entities["Nu"]=(wchar_t)0x039D;
entities["Xi"]=(wchar_t)0x039E;
entities["Omicron"]=(wchar_t)0x039F;
entities["Pi"]=(wchar_t)0x03A0;
entities["Rho"]=(wchar_t)0x03A1;
entities["Sigma"]=(wchar_t)0x03A3;
entities["Tau"]=(wchar_t)0x03A4;
entities["Upsilon"]=(wchar_t)0x03A5;
entities["Phi"]=(wchar_t)0x03A6;
entities["Chi"]=(wchar_t)0x03A7;
entities["Psi"]=(wchar_t)0x03A8;
entities["Omega"]=(wchar_t)0x03A9;
entities["alpha"]=(wchar_t)0x03B1;
entities["beta"]=(wchar_t)0x03B2;
entities["gamma"]=(wchar_t)0x03B3;
entities["delta"]=(wchar_t)0x03B4;
entities["epsilon"]=(wchar_t)0x03B5;
entities["zeta"]=(wchar_t)0x03B6;
entities["eta"]=(wchar_t)0x03B7;
entities["theta"]=(wchar_t)0x03B8;
entities["iota"]=(wchar_t)0x03B9;
entities["kappa"]=(wchar_t)0x03BA;
entities["lambda"]=(wchar_t)0x03BB;
entities["mu"]=(wchar_t)0x03BC;
entities["nu"]=(wchar_t)0x03BD;
entities["xi"]=(wchar_t)0x03BE;
entities["omicron"]=(wchar_t)0x03BF;
entities["pi"]=(wchar_t)0x03C0;
entities["rho"]=(wchar_t)0x03C1;
entities["sigmaf"]=(wchar_t)0x03C2;
entities["sigma"]=(wchar_t)0x03C3;
entities["tau"]=(wchar_t)0x03C4;
entities["upsilon"]=(wchar_t)0x03C5;
entities["phi"]=(wchar_t)0x03C6;
entities["chi"]=(wchar_t)0x03C7;
entities["psi"]=(wchar_t)0x03C8;
entities["omega"]=(wchar_t)0x03C9;
entities["thetasym"]=(wchar_t)0x03D1;
entities["upsih"]=(wchar_t)0x03D2;
entities["piv"]=(wchar_t)0x03D6;
entities["ensp"]=(wchar_t)0x2002;
entities["emsp"]=(wchar_t)0x2003;
entities["thinsp"]=(wchar_t)0x2009;
entities["zwnj"]=(wchar_t)0x200C;
entities["zwj"]=(wchar_t)0x200D;
entities["lrm"]=(wchar_t)0x200E;
entities["rlm"]=(wchar_t)0x200F;
entities["ndash"]=(wchar_t)0x2013;
entities["mdash"]=(wchar_t)0x2014;
entities["lsquo"]=(wchar_t)0x2018;
entities["rsquo"]=(wchar_t)0x2019;
entities["sbquo"]=(wchar_t)0x201A;
entities["ldquo"]=(wchar_t)0x201C;
entities["rdquo"]=(wchar_t)0x201D;
entities["bdquo"]=(wchar_t)0x201E;
entities["dagger"]=(wchar_t)0x2020;
entities["Dagger"]=(wchar_t)0x2021;
entities["bull"]=(wchar_t)0x2022;
entities["hellip"]=(wchar_t)0x2026;
entities["permil"]=(wchar_t)0x2030;
entities["prime"]=(wchar_t)0x2032;
entities["Prime"]=(wchar_t)0x2033;
entities["lsaquo"]=(wchar_t)0x2039;
entities["rsaquo"]=(wchar_t)0x203A;
entities["oline"]=(wchar_t)0x203E;
entities["frasl"]=(wchar_t)0x2044;
entities["euro"]=(wchar_t)0x20AC;
entities["image"]=(wchar_t)0x2111;
entities["weierp"]=(wchar_t)0x2118;
entities["real"]=(wchar_t)0x211C;
entities["trade"]=(wchar_t)0x2122;
entities["alefsym"]=(wchar_t)0x2135;
entities["larr"]=(wchar_t)0x2190;
entities["uarr"]=(wchar_t)0x2191;
entities["rarr"]=(wchar_t)0x2192;
entities["darr"]=(wchar_t)0x2193;
entities["harr"]=(wchar_t)0x2194;
entities["crarr"]=(wchar_t)0x21B5;
entities["lArr"]=(wchar_t)0x21D0;

[helios]

entities["uArr"]=(wchar_t)0x21D1;
entities["rArr"]=(wchar_t)0x21D2;
entities["dArr"]=(wchar_t)0x21D3;
entities["hArr"]=(wchar_t)0x21D4;
entities["forall"]=(wchar_t)0x2200;
entities["part"]=(wchar_t)0x2202;
entities["exist"]=(wchar_t)0x2203;
entities["empty"]=(wchar_t)0x2205;
entities["nabla"]=(wchar_t)0x2207;
entities["isin"]=(wchar_t)0x2208;
entities["notin"]=(wchar_t)0x2209;
entities["ni"]=(wchar_t)0x220B;
entities["prod"]=(wchar_t)0x220F;
entities["sum"]=(wchar_t)0x2211;
entities["minus"]=(wchar_t)0x2212;
entities["lowast"]=(wchar_t)0x2217;
entities["radic"]=(wchar_t)0x221A;
entities["prop"]=(wchar_t)0x221D;
entities["infin"]=(wchar_t)0x221E;
entities["ang"]=(wchar_t)0x2220;
entities["and"]=(wchar_t)0x2227;
entities["or"]=(wchar_t)0x2228;
entities["cap"]=(wchar_t)0x2229;
entities["cup"]=(wchar_t)0x222A;
entities["int"]=(wchar_t)0x222B;
entities["there4"]=(wchar_t)0x2234;
entities["sim"]=(wchar_t)0x223C;
entities["cong"]=(wchar_t)0x2245;
entities["asymp"]=(wchar_t)0x2248;
entities["ne"]=(wchar_t)0x2260;
entities["equiv"]=(wchar_t)0x2261;
entities["le"]=(wchar_t)0x2264;
entities["ge"]=(wchar_t)0x2265;
entities["sub"]=(wchar_t)0x2282;
entities["sup"]=(wchar_t)0x2283;
entities["nsub"]=(wchar_t)0x2284;
entities["sube"]=(wchar_t)0x2286;
entities["supe"]=(wchar_t)0x2287;
entities["oplus"]=(wchar_t)0x2295;
entities["otimes"]=(wchar_t)0x2297;
entities["perp"]=(wchar_t)0x22A5;
entities["sdot"]=(wchar_t)0x22C5;
entities["lceil"]=(wchar_t)0x2308;
entities["rceil"]=(wchar_t)0x2309;
entities["lfloor"]=(wchar_t)0x230A;
entities["rfloor"]=(wchar_t)0x230B;
entities["lang"]=(wchar_t)0x2329;
entities["rang"]=(wchar_t)0x232A;
entities["loz"]=(wchar_t)0x25CA;
entities["spades"]=(wchar_t)0x2660;
entities["clubs"]=(wchar_t)0x2663;
entities["hearts"]=(wchar_t)0x2665;
entities["diams"]=(wchar_t)0x2666;

[tummychow]

That's one lengthy table.

[helios]

Pfft. That's nothing. Check dis out:
http://onslaught-vn.svn.sourceforge.net/viewvc/onslaught-vn/trunk/src/SJIS.table.cpp?revision=121&view=markup
That's the compressed version. It originally looked like this (warning: VERY large):
http://onslaught-vn.svn.sourceforge.net/viewvc/onslaught-vn/trunk/src/SJIS.table.cpp?revision=94&view=markup

EDIT: Oh, just love this loop:

for (unsigned long b=/**/,c=/**/;c;a++,b++,c--){

[chrisname]

That loop is horrible. How does it even work? Surely you can't do an assignment like that?

[helios]

I removed the commented parts because it was too long:

void initialize_conversion_tables(){
    std::fill(SJIS2Unicode,SJIS2Unicode+0x10000,0x003F);
    std::fill(Unicode2SJIS,Unicode2SJIS+0x10000,0x003F);
    for (unsigned long a=0;;){
        if (!(SJIS2Unicode_compact[a] || SJIS2Unicode_compact[a+1]))
            break;
        a+=2;
        for (unsigned long b=SJIS2Unicode_compact[a-2],c=SJIS2Unicode_compact[a-1];c;a++,b++,c--){
            SJIS2Unicode[b]=SJIS2Unicode_compact[a];
            Unicode2SJIS[SJIS2Unicode_compact[a]]=(wchar_t)b;
        }
    }
}

[chrisname]

What is the point of this

for (unsigned long a=0;;){
        if (!(SJIS2Unicode_compact[a] || SJIS2Unicode_compact[a+1]))
            break;

?

[helios]

You mean of putting the condition outside the for? None, really, but what's the difference? At least it's slightly easier to read than
for (unsigned long a=0;SJIS2Unicode_compact[a] || SJIS2Unicode_compact[a+1];)

[chrisname]

I suppose so. What's with the names, anyway? What does "SJIS2" refer to?

for (unsigned long b=SJIS2Unicode_compact[a-2],c=SJIS2Unicode_compact[a-1];c;a++,b++,c--){
            SJIS2Unicode[b]=SJIS2Unicode_compact[a];
            Unicode2SJIS[SJIS2Unicode_compact[a]]=(wchar_t)b;

I hate having to do that with for loops and if statements. It makes me feel bad.

[helios]

"Shift JIS to Unicode" and "Unicode to Shift JIS". They're conversion tables.

[chrisname]

Ohh... I get it. That makes sense.