Recurrent Neural Network writes Semantic Web Specification

The other day danbri brought Recurrent Neural Networks to my attention, linking to the post Lisls Stis: Recurrent Neural Networks for Folk Music Generation. Now although I’ve always had one eye on AI stuff, I missed RNNs. They’re pretty amazing, check The Unreasonable Effectiveness of Recurrent Neural Networks.  I’ve been meaning to get back into signals a bit (FFT etc) especially in the context of neural nets, but despite having had a recent nosey around Octave (open source Matlab-like kit) hadn’t really bitten. You can throw signals at RNNs and they’ll learn stuff.

The tools mentioned behind the RNN links above are based on the Lua language, specifically using Torch (leading to a setup not unlike Octave/Matlab):

Torch is a scientific computing framework with wide support for machine learning algorithms. It is easy to use and efficient, thanks to an easy and fast scripting language, LuaJIT, and an underlying C/CUDA implementation.

Torch was relatively straightforward to set up (on Ubuntu), except I needed one tweak to the script install.sh – comment out -DWITH_LUAJIT21=ON (around line 41) before running.

There’s code for running an RNN on text on GitHub, char-nn.

For a play, I snagged the W3C semweb specs:

wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --restrict-file-names=windows \
     --domains www.w3.org \
     --level=1 \
         http://www.w3.org/2001/sw/Specs

(manually deleting the things that weren’t HTML spec docs)

Concatenated them together:

cat $(find . -name \*.html) > input.txt

producing a 17MB file, then passed to the RNN with the following:

th train.lua -data_dir data/w3c -gpuid -1 -rnn_size 128 -num_layers 3 -dropout 0.2  -max_epochs 20 -eval_val_every 200

3 layers, 128 nodes, pretty much guessed. I don’t have a suitable GPU (-gpuid -1) so set it running with nice=5 and left it to it. It quite quickly learnt the general shape of words and markup, though it was a few hours before it got remotely intelligible. It would have run for a week but crashed some time this morning, after 3 days (10 epochs), dunno why :

saving checkpoint to cv/lm_lstm_epoch10.91_1.3082.t7    
/home/danny/torch/install/bin/luajit: /home/danny/torch/install/share/lua/5.1/torch/File.lua:157: write error: wrote 206438 blocks instead of 446773 at /home/danny/torch/pkg/torch/lib/TH/THDiskFile.c:314

[PS. was out of disk space 🙁 ]

– but fortunately it was saving regular ‘checkpoint’ files so there was a recent one from which to generate some new material. I had to delete 3 little errors to get it through HTML Tidy so it wouldn’t mess up embedding here (I ignored the 96 warnings). Viewed directly in a browser the layout isn’t so scrunched.

So here is the result:

for all representations for expressed in an instance in
attribute languages version in the HTML has a subject extensions
contains a conditions,
in a starthor given encoding ovalue name. For example, the
association that derived truth mappings by not options. What ‘
OWL-RIF-Core Content )