Aigarius Blog

This is highly unofficial, but if you want to upload your World of Warcraft statistics to WoWHead in Linux, then you might be able to do so by using the following script. You will need curl and wget installed.

(Continued)

Popularity: 16% [?]

I am making a small project with TurboGears and I am just loving the widgets and the identity framework. There are some rough spots, like the documentation, but luckily for the most pars you can just launch an ipython interpreter and use tab completion to look at what functions are available. But then I got this error:
Traceback (most recent call last):
File "./start-foo.py", line 23, in
from foo.controllers import Root
File “…Foo/foo/controllers.py”, line 22, in
action=”save_settings”
File “/var/lib/python-support/python2.5/turbogears/widgets/meta.py”, line 169, in widget_init
validator = generate_schema(self.validator, widgets)
File “/var/lib/python-support/python2.5/turbogears/widgets/meta.py”, line 277, in generate_schema
if widget.is_named:
AttributeError: type object ‘SettingsFields’ has no attribute ‘is_named’

And I could not find any help on this. Luckily I found Lucas manual and by comparing the code I found that in this part of the code:
settings_form = widgets.TableForm(
fields=SettingsFields,
action="save_settings"
)


Instead of passing the whole SettingsFields class, an instance of it needs to be passed ( as in “fields=SettingsFields(),” ). Cann’t wait to see all the goodies they are making now in TG 2.0, but considering my experience, I will wait until they update the documentation for it before diving in.

Popularity: 21% [?]

I am not a good coder. In my mind a coder is determined by how much he loves C (or, in clinical cases, assembler). I hate C and avoid it whenever I can, because I am always off-by-one, and often even in more then one place, which make debugging even more .. fun.

Currently I am coding an application that is centred around use of artificial neural networks (more precisely a multilayer perceptron). The NN part must be done in C for speed and memory usage (imagine adding and multiplying hundreds of millions of floating values just for one operation .. and having to do that for 10 000 times for training .. while holding the whole data set in RAM, because you access every single value of it every second). With that in mind I engineered my system in such a way that the C part can be reduced to a couple simple operations that I can adapt from examples of the NN library. And still I barely wrote those pieces. Working with MySQL in C is pure pain, for example.

The rest of the system is in Python and I must say that Python is a godsend by itself, without even comparing it to C. I am now so used to Python when I was debugging my C program, I automatically wrote a chunk of Python into it and was very surprised when it did not work.

With that in mind, to ease the pain for further generations of neural network researchers, I will (someday) find some time and write Python wrappers for the nice and cool Lightweight Neural Network C library that I am currently using here.

BTW: I found that SourceForge is packed full with great software projects, but it is very hard to find those projects in Google. Morale - seeking for good free software? wajig (or apt-get) it! then sourceforge it! then freshmeat it! and only then google it.

Popularity: 34% [?]

As I decided to migrate my blog from blogger.com to my own blogging software on my own domain, I did not want to leave my old posts behind. As I had more then 150 of them, I had to do some automation to do that within my lifetime.

The first step is to get the data from Blogger.com database to somewhere where you can access it in full. You will need FTP or SFTP access to a computer with a real IP address. The idea is to go to blogger.com and specifiy that you want to switch storage of your blog pages from their blogspot.com service to your of FTP or SFTP server. You will have to provide server address, URL, folder, username and password. URL does not matter at this point, but other parameters define where your data will be dumped to.

Save the changes and republish the blog. After a few dozens of minutes all your data will be at your server in a set of complete html files. Now we need something that will parse those html files and write out something that your blogging software can understand. As I use Mnemosyne, my format is a set of mail message files.

To allow me to use HTML formatted messages in my blog (and not only reST) I added this to my config.py:

 class EntryMixin:    def _init_content(self):        """Read in the message's body, strip any signature, and format using        reStructedText unless X-Format=='html'."""         s = self.msg.get_payload(decode=True)        if not s: return ''         try: s = s[:s.rindex('--  ')]        except ValueError: pass         body=False         try:                if self.msg['X-Format'] == "html":                        body = s.replace("&nbsp;", " ")                        body = re.sub(r'&(?!\w{1,10};)',r'&amp;',body)                        body = xml.dom.minidom.parseString("<div>"+body+"</div>").toxml()        except KeyError:                pass        except xml.parsers.expat.ExpatError, e:                print "W: Parse failed for "+self.msg['Subject']+" at "+self.msg['Date']+" from "+str(int(time.mktime(time.strptime(self.msg["Date"],"%a %b %d %H:%M:%S %Y"))))                print xml.parsers.expat.ErrorString(e.code), e.lineno, e.offset         if not body:                parts = docutils.core.publish_parts(s, writer_name='html')                body = parts['body']         self.cache('content', body)        return body 

This will try to parse messages as pure xHTML if custom header “X-Format” is set to “html” in the blog entry. There is one problem with this approach - xHTML must be valid, otherwise XML parser in Kid templating engine will fail and there will be no end of trouble. That is why even in HTML mode we reparse the body to XML and back to string again. If we get a parsing error at that point, we fall back to reST parser.

Now we need something that will analyse our Blogger.com generated HTML files and get our content from there. Here is the script that I used:

 #!/usr/bin/python  import os, os.path, time, sys, glob, re, xml.dom.minidom id = 1 host = "old"  mdate = re.compile( r'\">(\d+) (\D+) (\d+)<' ) mbody = re.compile( r'</div>(.+)<div style=' ) mfooter = re.compile( r'<a href=\"http://example.com/(.+)\" title=\"permanent link\">(\d+):(\d+)</a></em>' )  files = glob.glob("old/*/*/*.html")  for file in files:        year = 0        month = 0        day = 0        hour = 0        minute = 0         subject = ""        oldurl = ""        body = ""        date = ""        title = ""        footer = ""         f = open( file, "r" )        status = 0        for l in f:                if status == 0:                        if l.find('class="date-header"')>0:                                date = l                                status = 1                elif status == 1:                        if l.find('class="post-title"')>0:                                status = 2                        if l.find('class="post-body"')>0:                                status = 4                elif status == 2:                        if len(l.strip()) > 0:                                title = l                                status = 3                elif status == 3:                        if l.find('class="post-body"')>0:                                status = 4                elif status == 4:                        if len(l.strip()) > 5:                                body += l                        if l.find('padding-bottom: 0.25em;')>0:                                status = 5                elif status == 5:                        if l.find('posted by ')>0:                                footer = l                                break         f.close()         rdate = mdate.search( date )        rbody = mbody.search( body )        rfooter = mfooter.search( footer )         year = rdate.groups()[2].strip()        month = rdate.groups()[1].strip()        day = rdate.groups()[0].strip()         subject = title.strip()         body = rbody.groups()[0]        body = "<p>"+body+" </p>"        body = re.sub(r'<img([^>]*?[^/])>',r'<img\1/>',body)         if subject == "":                subject = re.sub(r'<br.*?>',' ', body)                subject = re.sub(r'</p.*?>',' ', subject)                subject = re.sub(r'<.*?>','', subject)                subject = subject.strip()                line = subject.find(' ')                if line > 45:                        subject = subject[:40]+"..."                else:                        subject = subject[:line]          oldurl = "http://oldblog.blogspot.com/"+rfooter.groups()[0].strip()        hour = rfooter.groups()[1].strip()        minute = rfooter.groups()[2].strip()         mtime = time.mktime(time.strptime(day+" "+month+" "+year+"  "+hour+":"+minute, "%d %B %Y %H:%M"))        outname = str(int(mtime))        while os.path.exists( outname+"."+str(id)+"."+host ):                id += 1        outname = outname+"."+str(id)+"."+host        out = open( "entries/new/"+outname, "w" )         out.write("Date: "+time.ctime(mtime))        out.write(" Subject: "+subject)        out.write(" X-URL: "+oldurl)        out.write(" X-Tags: untagged")        out.write(" X-Format: html")        out.write("  ")        out.write(body)        out.close() 

Here I parse the HTML files using a simple state automata and then assemble all the data that we need to have, like timestamp in filename and Date field. In this script it is assumed that in the current directory there is “old/” directory with subdirectories like “2005″, “2006″, … which have subdirectories for months in which there are xHTML files for individual posts. The output is written to “entries/new/$timestamp.$id.$host” files.

After running this script and then running mnemosyne you will see a bunch of messages about failed parsing of some messages. That shows you where your xHTML is not valid. The usual problems are html entities ( like € ) that the parser does not recognise and lack of closure on img or br tags:

 This will fail because of &euro;  <img src="smile.png"> will fail, it must be <img src"smile.png"/> 

My import script fixes img and br tags, but for other problems there is not much choice but going trough the entries and fixing them up manually. Also later on, if you will want to paste some custom HTML into your post, you will have to mark the whole post as HTML mode post and also check if you pasted HTML is valid XML manually.

There are some other fun things in this blog, but I will go into that in later posts.

Popularity: 29% [?]

Popularity: 18% [?]

Popularity: 25% [?]

Popularity: 22% [?]

Popularity: 19% [?]

Popularity: 16% [?]

Popularity: 28% [?]