Monthly Archive: September, 2017

tldr; When parts of sites are generated on the fly by browser Javascript, they can’t be directly archived. Augmenting a simple crawler with an easy-to-use browser automation subsystem (Selenium) makes this very straightforward. Background If you got past the title of this post, chances are you’ve executed a command something like this: wget –mirror –convert-links –adjust-extension –page-requisites –no-parent http://example.org This...

I noted recently with my ELFquake project that, at least with a slow brain like mine, there’s a definite efficiency trade-off between procrastinating and just Doing Stuff. In the time it took me to get around to implementation, research and plain old thought had led me to much better, quicker ways of doing things. I’ve just had another episode of...