I'd Rather Hack

Movgrab Popularity at 400

2011-12-21T02:53:00.001-08:00

Movgrab's freshmeat (they're changing their name to 'freecode' but you'd know them as freshmeat if you know them at all) popularity rating has passed the '400' mark. I would like to say that it's smashed through this barrier, but as it's just managed to struggle to 400.13, and will probably drop back below 400 soon, well, I can't really say that.

I've no idea what this number really means, but I think 400 means 'quite popular'. Movgrab is now included in a few linux distributions, like Puppy linux, Arch linux and maybe Debian (someone did contact me asking me to make a change to allow it to be included in the debian build process).

One person has even written a nice-looking frontend to movgrab, I think this frontend comes as part of puppy linux.

You wouldn't think it from the feedback I've had though, despite setting up an email account for people to contact me with bug reports etc, very, very few people have done so. If you're one of those who did (you know who you are) then thanks very much, some of the bug reports I got were things that I'd never have known about it people hadn't taken the time to report them.

Now it's time to get some other projects finished and uploaded!

Using unix consolefonts in your programs

2011-11-10T09:43:00.001-08:00

If you're running linux or another unix-style system, then you probably have a little-known directory in '/usr/share/consolefonts'.

This contains 'font files' that can be used with the 'setfont' command to change the default font at the linux console (not on xterm or rxvt or anything like that, at the actual 'text mode' console). If you're looking for how to do that, it's as simple as:

setfont drdos8x16
setfont cybercafe

where the 'name' of the font is it's filename without path or extensions (setfont knows to look in /usr/share/consolefonts).

This post is about how to decode these font-files for use in your own graphics programs.

These fonts have a very simple format, being little more than a collection of bitmaps (not .bmp format, but actual bitmaps, i.e. collections of bytes where every bit is a monocrhome 'pixel' that's on or off). If you're messing about with low-level graphics, they are an easy way of doing text.

There's 3 types of font in the console font directory.

1: Raw Fonts

These usually have the file extension '.fnt' (or .fnt.gz if gzipped). They contain an array of 8x16 bitmaps, or 'glyphs' of the characters in the font. The bitmaps are arranged so that they map to the ascii character set, so to find the bitmap for ascii character 'A' which has an ascii value of 65, you would just get the 65th bitmap in the array. As each character is 8x16, and so 16 lines of 8 bits, or 16 bytes, so you just calculate the position in the bitmap array as '<ascii value> * 16'.

Thus, the first byte of the file is the first 'line' of 8 pixels in the first character. For each bit that is set in those eight bytes, you should set or 'color' a pixel in an image, and leave blank those pixels that match to unset bits in the byte. The second byte is the second line of the character, and so on.

2: PSF Fonts

These are very similar to 'Raw' fonts, except they have a header. This header starts:

Byte 0: 0x36
Byte 1: 0x04
Byte 2: 'Mode' (256 characters or 512 characters)
Byte 3: Character Height

The first two bytes are just for identifiying the file type, and the 'Mode' byte says how many characters are stored in the font-file. If the first bit of this byte is set, then there are 512 characters, otherwise it's 256. The final header byte is the character height, allowing you to have glyphs that are 8x8, or 8x12 or 8x128, or whatever. Notice though, that for basic PSF fonts, the width of the character is still always 8 bits, or one byte.

After this 4-byte header the font is just the same as 'raw' fonts, except you have to consider that each glyph is as many bytes long as the 'Character Height' setting says.

3: PSF2 Fonts

The most advanced type of console font, it seems, are an extended version of 'PSF' fonts, called 'PSF2' fonts. These have an enhanced header:

Byte0: 0x72
Byte1: 0xb5
Byte2: 0x4a
Byte3: 0x86
unsigned int version;
unsigned int headersize; /* offset of bitmaps in file */
unsigned int flags;
unsigned int length; /* number of glyphs */
unsigned int charsize; /* number of bytes for each character */
unsigned int charheight;
unsigned int charwidth;

After the first 4 identifying bytes all the information is unsigned 4-byte integers. Everything does pretty much what it says. As now both the character width and character height can change, so you can no-longer calculate glyph positions in the file with 'lines * NoOfLines', but you can use the helpful 'charsize' value. Thus the glyph bitmap for a given character is found at 'headersize + <ascii value> * charsize'. 'length' tells you how many characters are in the file, and 'charwidth' now means you have to consider that each line of a character might use less than one byte, or stray across two or more bytes. However, the bitmap data is padded to a byte boundary, so if a font has a width of '12', or 'a byte and a half' it will still use 2 bytes for every line in every character.

Some of the default fonts distributed with the linux 'kbd' package (arm8.fnt, cybercafe.fnt) look surprisingly good when used on charts and the like, and they're about the easiest font-type to use in your own programs without turning to libraries like 'Freetype'.

Howto use Picasa API with Blogger API

2011-11-09T03:26:00.001-08:00

When you use the blogger api, the pages you get back contain a type of tag called 'gd:extendedProperty'. These contain some useful data.

Just do an http get to: <blogurl>/feeds/posts/default

So, for this blog you are reading: http://idratherhack.blogspot.com/feeds/posts/default

This will return a big mess o' xml, including the tags:

<gd:extendedProperty name='IS_PUBLIC_BLOG' value='true'/>
<gd:extendedProperty name='PICASAWEB_ALBUM_ID' value='5570923802870618657'/>

It turns out that one of these (PICASAWEB_ALBUM_ID) is a link to the Picasa album that is used by your blog.

You can use this link to upload pictures using the picasa api documented here http://code.google.com/apis/picasaweb/docs/2.0/developers_guide_protocol.html.

When you do the upload you'll get back another mess-o-html which will include a tag called 'content'. This should give you the 'web visible' link for the image, which you can now use in posts to your blog.

Can you do Rocket Science?

2011-10-18T02:35:00.001-07:00

I'm cross-posting this to my 'technical' blog, because I think that it might attract a few people who write non-fiction, and that while most of those will be a 'computery' people, a few might be rocket scientists.

Can you write non-fiction on technical space-related matters, or on aspects of 'Hard SF'? Would you like to be published in an upcoming anthology?

Ian Sales' Rocket Science anthology coming out from Mutation Press (that's the award-winning Mutation Press) hasn't had enough non-fiction submissions, he says he's "Desperate" for them.

He's also had nothing like as many submissions from women as he'd like, either of fiction or non-fiction.

'Call for Submissions' is here

Know something about rockets/space/hard sf? Want to tell the world about it? This is your chance.

New release of my 'getlock' program

2011-10-07T01:50:00.001-07:00

A new version of my 'getlock' locking program is out. I use this to ensure that scripts and programs running out of 'cron' don't start running multiple copies that 'fight' each other. getlock creates a lock file and locks it with fcntrl locking, and then runs the program or script, keeping the file locked for the whole time that program runs. The advantage of this scheme over other locking programs is that if the 'getlock' process is killed or otherwise stops the lock is released automatically.

'getlock' has all sorts of 'timeout' features regarding how long it waits for a lock, and whether it 'steals' a lock from a program that's currently holding it, etc, etc.

The new version has the added feature of an 'abandon timeout', this is a timeout that applies to the running program. If the program has 'frozen' for some reason, then the 'abandon timeout' can be used so that getlock can detect the program is taking too long, and kill it off and exit. I use this, for instance, in situations where a script or program connects to hosts or websites that might become unresponsive, leaving the program waiting for network data that will never come. This can result in a 'hung' program that's holding the lock file and preventing another instance of the process (which will reconnect and start from scratch, and may thus succeed) from starting up.

Get it here:

http://sites.google.com/site/columscode/files/getlock-0.2.tar.gz?attredirects=0

Disable Screen Blanking in Linux: How hard can it be?

2011-09-05T01:38:00.001-07:00

How hard can it be? Well, prepare to be amazed, I've spend days trying to figure out how to turn off screen blanking under linux, and the internet is full of people bewailing the same issue.

I'm working on a job that has a 'management' screen that's permanently displaying information to any passing users. Unfortunately screen blanking keeps kicking in, and as the screen isn't supposed to have a keyboard or mouse, this is proving to be a problem.

The main cause of difficulty is that there are several different systems that can blank the screen. Firstly, there's the linux console, this is the 'text mode' system that's operational even when you're not running a graphical display (you probably knew that).

Once upon a time, this would have been sufficient to turn off screen blanking at the console:

setterm -blank 0

However, modern monitors can powerdown (which old-skool CRT's generally didn't, as in those days fossil fuel resources grew on trees. But then we cut all the trees down). Powering down is different to blanking the screen, it's actually turning the screen off, so you also need to do

setterm -powerdown 0

But there is also 'powersave'. I don't know how this differs from powerdown, but you should also do

setterm -powersave 0

So that's three commands needed for you to be sure that you've switched off screen blanking/turning off in the console alone, before we've even started X.

Incidentally, the 'setterm' command has many interesting features, including the ability to change the color of the screen text and background, take a textmode 'screen dump', turn on blink, underline, reverse, bold, linewrap, the cursor, etc, etc. This means it can be used in shell scripts to output text in different colors or bold or blink.

Now, we start the dreaded 'X', and enter his (yes, 'X' is definitely male) dark cabinet of madness.

X has its own screensaver, that defaults to blanking the screen if no screensaver is set up. You can check the status of this by running the command:

xset -q

This will print out the status of the various things that 'xset' can mess with. One of them will be:

Screen Saver:
prefer blanking: yes allow exposures: yes
timeout: 1200 cycle: 600

If timeout is non-zero, then the screen saver will kick in after that many seconds. Turn it off with:

xset s off

But this still isn't the end of it. There's one more thing that can powerdown/blank your screen. It's called 'DPMS', and I think it relates to the monitor's own ability to power itself down independent of the computer using it. If 'xset -q' shows:

DPMS (Energy Star):
Standby: 1200 Suspend: 1800 Off: 2400
DPMS is Enabled

Then again, the monitor is going to go into low-power modes (including 'Off') after the time has elapsed. This final hurdle can be overcome with:

xset -dpms

And now, with luck, you should find that your screen doesn't blank. Maybe.

Stuff that Works Under Linux: Wifi

2011-07-02T07:17:00.001-07:00

Finding hardware that works with linux can STILL be a frustrating experience even after all these years. Manufacturers persist on focusing on 'Windows and mac'. This makes some sense nowadays when apple has come back from the brink and everyone has an macbook (or whatever they call them now) but it was really weird ten years ago. Ten years ago NO ONE owned apple products, as evidenced by the way that a mac in a public place would gather a crowd of wary onlookers, rather like the crowd that might surround a crashed alien starship once they'd got over the initial panic and the joy of looting had begun to pale.

I used to complain 'More people use linux than macs, so why do manufacturers keep supporting this dying platform, and ignoring linux?'

To which people would say: "Oh no, lots of people use macs"

To which I would respond, "Oh yeah? Anyone you know?"

People: "Well yes, sure."

Me: "Who?"

People: "Oh lots of people we know use macs. Lots."

Me: "Name one."

People: "Uh, well, there's... um... Jim?"

Me: "Jim? That's funny, you've never spoken of him before, he mustn't get out much. Does graphic design, does he?"

People: "Well yes, that's why he needs a mac. Uh-huh. Yes. Honest."

These mysterious 'Jims' and 'Sams' and 'Sues' always did graphic design, leading me to claim that it was the same guy, who just got around a lot (and sometimes wore a skirt).

Apple has always enjoyed some weird must-support-the-underdog loyalty from the public, evidenced by the fact that, in the days when the company was hanging by its fingernails over the abyss of chapter-11 bankruptcy BECAUSE IT WASN'T SELLING ANYTHING, and Michael Dell was saying they should just kill it, asset-strip the corpse, and give the money to the shareholders, people were prepared to lie and invent pretend friends in order to claim they knew someone who used macs. No-one used macs. Most people had never even seen one.

But times have changed and now the mac has become so ubiquitous that I recently went to a Linux User Group, and everyone had macs (Huh? Isn't that like going to a church service and finding everyone is wearing t-shirts that say "SATAN IS LORD!!"?). But still, I continue to wonder why linux is so badly supported.

Part of the problem, I think, is corporate attitudes to open-sourcing their driver code/protocols. Now, I've always thought this was odd. Back in the day I remember that large numbers of network cards didn't work under linux, because vendors wouldn't make information available about how to talk to the card. This stuff was all classified 'company secrets'.

Which leads one to wonder what the heck was in those cards? If the cards allowed you to signal to arbitrary points in space-time, or had a secret setting that caused them to print money (erudite readers will realise these two features amount to the same thing) then I could understand this, but they didn't, they just sent messages at the industry-standard speed down a wire. This means that the big secret that couldn't be revealed must have consisted of an interface of the form:


int SendPacket(char *Destination, char *Data, int Length);
int RecvPacket(char *Source, char *Data, int Length);

Sorry, but that's not world-changing technology that anyone's going to steal! WE ALREADY HAVE THAT!

My suspicion was always that what they were hiding was BAD DESIGN. Coding practices and protocol design so horrifically deranged that they could have a walk-on part in an H.P. Lovecraft story.

Hmm, I'm rambling rather badly here, let's pull things back to the point. The new equivalent of those network cards (whose vendors all went out of business, I think. Or rather, I hope) are Wifi cards. There's loads of them, and only a few of them seem to work with linux.

Indeed so bad is the support for wifi cards in linux that someone's come up with something called ndiswrapper that lets you use windows wifi drivers under linux. I tried this, but the very first sign that we weren't going to get on, was that it required me to install perl. I don't want to install perl just to use a piece of hardware/COMPILE A KERNEL/send an email/etc/etc/etc any more than I want to install Python to be able to compile X, gnome or KDE to play old arcade games or browse the web, or install windows to be able to use my computer. But this is another rant. Anyways, this time I gave in, and installed the huge, bloated, misshapen camel-monster that is perl onto my (previously) small and slick system. After doing that, and installing ndiswrapper and going through the time-consuming business of extracting the windows drivers and installing them, everything functioned as expected.

Meaning it DIDN'T F****ING WORK!

So I ripped all that stuff back off my system (though I'm sure there's a few camel-colored tentacles left twitching around in there somewhere) and went out and bought some hardware that's supported by the vanilla linux kernel.

Anyways, below is a list of some wifi stuff that I found works under linux without horrible camel-powered bodges. Many of these devices will be old technology, because I'm the kind of person who likes to find use for the things that the ~~surplus sheeple~~ everyday folks cast aside.

One interesting note: If found that stuff that works under linux, also works under windows, and b(stuff that doesn't work under linux, won't work under windows next year). I have a lot of old technology come my way, and when I find a card or dongle that doesn't work under linux, I try to give it to one of the windows-using relatives who come to me asking if I've got a wifi thing they can use. They come back to me telling me that they couldn't get it to work, and promptly steal one of my linux-compatible devices, and later report delightedly that it 'just worked'. This confirms something I've long said, b(if it doesn't work under stock linux, it's junk). It might work under windows right now, but in a year the company will have folded, you won't be able to get drivers any more, or there'll be a new version of windows under which it doesn't work. Buying cheap tech often means buying cheap tech five times over, where paying double means you get something that lasts five times as long. Do the math.

Right, onto the devices.

SAFECOM SWMULZ-5400

These are great, they're the little wifi-dongle that could. Provided you have Zydas zd-1211b chipset support compiled into your kernel, they just work. This model supports 54g wifi speeds, supports 'ad-hoc' mode as well as the usual 'managed' mode, and seems to have moderately good range. Just about any wifi dongle with the same zd-1211b chipset should work under linux without needing ndiswrapper or any special treatment. Here is a page listing devices that have this chipset. So good are these, that I carry one in about in a pocket, because you never know when someone's going to say "hey, my wifi's stopped working", or something like that.

WIFI-SKY IDU-2850UG-10G

If you've got reception issues, this is the beast for you. It uses the adopts Realtek RTL8187+RTL8225-VF chipset, but unfortunately the in-kernel linux driver doesn't seem to support ad-hoc mode, nor does it support the 'high transmit power' feature of the device. However you can stick and stonking great aerial on the thing (it comes with a good one), and this improves communications considerably. It also looks cool. Supports speeds up to 56g.

TP-LINK TL-WN610G

If you're using PCMCIA, I've found this is a nice, basic card. My one of these seems to have started playing up recently, but that's only after many years of daily service, so I'd still recommend the card. It supports speeds of up to 108 Meg 'super-g', but I think this requires a special access point, so realistically it's 54g like everything else.

NETGEAR MA401

Okay, this is an old, old, old card, it's going to be hard to find these days. But it just works. It only does 11b speeds. But it works. And it keeps working. Mine's been dropped, sat on, stepped on and is now held together with large blobs of superglue, but it works. Old tech tends to work, and keep working, whereas newer stuff is cost-engineered to extract the maximum money from your pocket while providing the most substandard parts possible, and designed to mortally fail a week after you warranty runs out. Buy the old stuff, it can take a kicking and keep on ticking.

Don't, however, assume that this one good card means that netgear products generally work with linux. My experience is that they're very hit-and-miss.

PLUSCOM WP-RT2561T

It does seem that if you see 'Ralink chipset' on a product, you're in with a good chance of having it work under linux. This pluscom card uses the Ralink 'RT2561' chipset.

Okay, that's the good, now for the bad and the ugly.

Belkin products

YOU PAYS YOUR MONEY; YOU TAKES YOUR CHANCE I have a number of Belkin F5D7050uk USB dongles, and they work very well under linux. But this is another of those devices where you have to be very sure you've got the right version of the device. My units are 'Version 3' (I think), and use the Ralink chipset, so they work under linux. However, other versions use a zydas, intersil or realtek chipset. These all look the same, and unless you're buying from ebay with a vendor who states the chipset, you're taking a gamble as to which version you get. I think all these versions have chipsets that should work under linux, but I don't know it for sure, so I can't really recommend them.

I've gotten other belkin hardware, like PCI wifi cards, to work under linux, but I've often found it's a real battle. I'd put belkin stuff under the 'ugly' category.

Linksys products:

AVOID My experience is that they flat-out don't work under vanilla linux kernels.

D-Link "Air Plus G" and other products using the Texas Instruments 'Acx100' chipset

AVOID I used a D-Link DWL-640G+ for some years, but the driver never got into the mainline kernel, and further development appears to have now been abandoned. The drivers probably won't even work with recent kernels, so these cards are scrap.

And that's all I've got.

Movgrab Status

2011-06-22T00:47:00.001-07:00

Current status output of movgrab ('movgrab -test-sites'). This shows which downloader sites are broken, and which I can't figure out, or which seem to have shut down. Long-term broken sites will eventually get removed from the list.

Checking youtube okay
Checking metacafe BROKEN
Checking dailymotion okay
Checking break okay
Checking ehow okay
Checking vimeo BROKEN
Checking almostkilled okay
Checking 5min okay
Checking vbox7 BROKEN
Checking blip.tv okay
Checking ted okay
Checking myvideo okay
Checking clipshack okay
Checking mytopclip okay
Checking redbalcony okay
Checking mobango okay
Checking berkeley okay
Checking yale okay
Checking sdnhm okay
Checking uchannel BROKEN
Checking princeton okay
Checking uctv.tv okay
Checking reuters okay
Checking clipfish.de okay
Checking liveleak okay
Checking academicearth okay
Checking photobucket okay
Checking videoemo okay
Checking videosfacebook okay
Checking aljazeera okay
Checking mefeedia okay
Checking myvido1 okay
Checking iviewtube okay
Checking washingtonpost okay
Checking cbsnews okay
Checking france24 okay
Checking euronews okay
Checking metatube okay
Checking motionfeeds okay
Checking izlese BROKEN

Movgrab at 300 popularity level on freshmeat

2011-06-21T22:25:00.001-07:00

Hurrah!!

My video-downloader program 'movgrab' passes the meaningless but psychologically important level of 300 in its popularity score on freshmeat.

That makes it nearly as popular as Bitcoin

It's the most popular I've ever been!

Three URL Shorteners that don't need authentication, XML or JSON

2011-06-19T16:51:00.001-07:00

Here's three URL shorteners that will allow you to get a shortened link without needing authentication, and with very simple APIs. These are useful if you're writing blogging or microblogging (twitter-style) applications.

All three take a simple HTTP GET request. As you have to include the URL you want shortened in the request, you have to HTTP encode it (you know, all that %20 stuff). All of them return one line of text, which is the shortened version of the URL.

You should be able to test them by simply using a webbrowser and going to the link. Don't forget to replace the line phrase <url goes here> with an http-encoded url.

Cli.gs http://cli.gs/api/v1/cligs/create?appid=autoblog-0.1&url=<url goes here>

Linkee http://api.linkee.com/1.0/shorten?format=text&input=<url goes here>

Is.gd http://is.gd/create.php?format=simple&url=<url goes here>

Howto Change Blogger Settings with the GData API

2011-06-18T08:11:00.001-07:00

I've just discovered an undocumented (or I couldn't find any docs for it anyways) way to change the settings of your Blogger/Blogspot blog via the GoogleData API.

I discovered this as a result of the recent 'breakage' of my API client. I was using a blog-posting 'endpoint' URL that wasn't supposed to work, but had been working until recenty. That was a url of the form:

http://idratherhack.blogspot.com/feeds/posts/default

But apparently I should have been looking up a special url of the form:

http://www.blogger.com/feeds/8441087078477003568/posts/default

Where the long number is a unique ID for my blog.

To get this special url, you have to first authenticate, and then do a GET to:

http://www.blogger.com/feeds/default/blogs

This will return a big ol' mess o' XML:


<?xml version='1.0' encoding='UTF-8'?><?xml-stylesheet href="http://www.blogger.com/styles/atom.css" type="text/css"?><feed xmlns='http://www.w3.org/2005/Atom' xmlns:openSearch='http://a9.com/-/spec/opensearch/1.1/' xmlns:gd='http://schemas.google.com/g/2005' gd:etag='W/&quot;A04NRno-fip7ImA9WhZbE0Q.&quot;'><id>tag:blogger.com,1999:user-04780753778872137937.blogs</id><updated>2011-06-18T12:26:37.456Z</updated><title>Colum's Blogs</title>
<link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='http://www.blogger.com/feeds/04780753778872137937/blogs'/>
<link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/04780753778872137937/blogs'/>
<link rel='alternate' type='text/html' href='http://www.blogger.com/profile/04780753778872137937'/>
<author>
<name>Colum</name>
<uri>http://www.blogger.com/profile/04780753778872137937</uri>
<email>noreply@blogger.com</email>
</author>
<generator version='7.00' uri='http://www.blogger.com'> Blogger</generator>
<openSearch:totalResults>2</openSearch:totalResults>
<openSearch:startIndex>1</openSearch:startIndex>
<openSearch:itemsPerPage>25</openSearch:itemsPerPage>
<entry gd:etag='W/&quot;DUICSH49cSp7ImA9WhZbE0Q.&quot;'>
   <id>tag:blogger.com,1999:user-04780753778872137937.blog-8441087078477003568</id>
   <published>2009-11-29T11:35:11.006-08:00</published>
   <updated>2011-06-18T04:46:09.069-07:00</updated>
   <title>I'd Rather Hack</title>
   <summary type='html'>Being the technical blog of Colum Paget, rather than his 'ranting into the void' one.</summary>
   <link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/04780753778872137937/blogs/8441087078477003568'/>
   <link rel='alternate' type='text/html' href='http://idratherhack.blogspot.com/'/>
   <link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='http://idratherhack.blogspot.com/feeds/posts/default'/>
   <link rel='http://schemas.google.com/g/2005#post' type='application/atom+xml' href='http://www.blogger.com/feeds/8441087078477003568/posts/default'/>
   <link rel='http://schemas.google.com/blogger/2008#template' type='application/atom+xml' href='http://www.blogger.com/feeds/8441087078477003568/template'/>
   <link rel='http://schemas.google.com/blogger/2008#settings' type='application/atom+xml' href='http://www.blogger.com/feeds/8441087078477003568/settings'/>
   <author>
   <name>Colum</name>
   <uri>http://www.blogger.com/profile/04780753778872137937</uri>
   <email>noreply@blogger.com</email>
   </author>
   <gd:extendedProperty name='IS_PUBLIC_BLOG' value='true'/>
   <gd:extendedProperty name='PICASAWEB_ALBUM_ID' value='5570923802870618657'/>
</entry>
<entry gd:etag='W/&quot;C08NR3oyeyp7ImA9WhZUGEQ.&quot;'>
   <id>tag:blogger.com,1999:user-04780753778872137937.blog-8531648592648053901</id>
   <published>2009-10-19T18:36:24.631+01:00</published>
   <updated>2011-06-12T16:18:16.493+01:00</updated>
   <title>The Singularity Sucks</title>
   <summary type='html'> The 21st century doesn't live up to the promises made when I sold my soul to SF back in the 70's, and I want my money back.</summary>
   <link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/04780753778872137937/blogs/8531648592648053901'/>
   <link rel='alternate' type='text/html' href='http://thesingularitysucks.blogspot.com/'/>
   <link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='http://thesingularitysucks.blogspot.com/feeds/posts/default'/>
   <link rel='http://schemas.google.com/g/2005#post' type='application/atom+xml' href='http://www.blogger.com/feeds/8531648592648053901/posts/default'/>
   <link rel='http://schemas.google.com/blogger/2008#template' type='application/atom+xml' href='http://www.blogger.com/feeds/8531648592648053901/template'/>
   <link rel='http://schemas.google.com/blogger/2008#settings' type='application/atom+xml' href='http://www.blogger.com/feeds/8531648592648053901/settings'/>
   <author>
   <name>Colum</name>
   <uri>http://www.blogger.com/profile/04780753778872137937</uri>
   <email>noreply@blogger.com</email>
   </author>
   <gd:extendedProperty name='IS_PUBLIC_BLOG' value='true'/>
   <gd:extendedProperty name='PICASAWEB_ALBUM_ID' value='5397308624374094369'/>
</entry>
</feed>

Pretty gnarly, huh? It's even worse than this, I've cut a lot of stuff out.

Notice that in all that guff, there are two <entry> um... entries, that contain the information for my two blogs. Some of that information is a bunch of 'link' entries, that tell you the URLs (or 'endpoints') to use for certain REST operations (like posting a blog entry). Each has a 'rel' parameter that specifies what the url is for.

For posting blog entries, you should be using the one marked 'rel=http://schemas.google.com/g/2005#post', (which in this example is 'http://www.blogger.com/feeds/8531648592648053901/posts/default')

Howerver, there's two others that are mysterious to me, but interesting looking. These are the ones marked

rel='http://schemas.google.com/blogger/2008#template'
rel='http://schemas.google.com/blogger/2008#settings'

What can they be for?

Well, doing a GET to http://www.blogger.com/feeds/8531648592648053901/settings produces a vertiable deluge of XML. There's far too much of it to post here, but it's made up of entries like:


<entry gd:etag='W/&quot;C0QDSHo4eCp7ImA9WhZbE0U.&quot;'>
<id>tag:blogger.com,1999:blog-8441087078477003568.settings.BLOG_ADMIN_PERMISSION</id>
<published>2009-11-29T11:35:11.006-08:00</published>
<updated>2011-06-18T00:16:19.430-07:00</updated>
<app:edited xmlns:app='http://www.w3.org/2007/app'>2011-06-18T00:16:19.430-07:00</app:edited>
<category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/blogger/2008/kind#settings'/>
<title>The list of administrators' emails for the blog.</title>
<content>admin@googlemail.com</content>
<link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/8441087078477003568/settings/BLOG_ADMIN_PERMISSION'/>
<link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/8441087078477003568/settings/BLOG_ADMIN_PERMISSION'/>
</entry>

Each entry refers to one setting related to your blog. The 'title' tags enclose a description of what the setting is, and the 'content' tags enclose the value of the setting.

Notice that there's a 'link' tag with a parameter of 'rel=edit'. Normally these 'edit' links can be used to send data to a google service. I wondered if this link could be used to change the settings related to my blog?

Turns out that yes, they can.

Doing an HTTP PUT to the 'edit' url of the setting thusly:


PUT /feeds/8441087078477003568/settings/BLOG_BACKLINKS_ALLOWED HTTP/1.1
Host: www.blogger.com
Content-Type: application/atom+xml
Content-Length: 342
Connection: Close
GData-Version: 2.0
Authorization: GoogleLogin auth=<Your auth token goes here>

<?xml version='1.0' encoding='UTF-8'?><entry xmlns='http://www.w3.org/2005/Atom' xmlns:gd='http://schemas.google.com/g/2005'><category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/blogger/2008/kind#settings'/><id>tag:blogger.com,1999:blog.settings.BLOG_BACKLINKS_ALLOWED</id><content>true</content></entry>

Allows you to change a setting. You should get a 200 OK response if it worked.

Notice that you need to have this an id tag like this:

<id>tag:blogger.com,1999:blog.settings.BLOG_BACKLINKS_ALLOWED</id>

Basically change 'BLOG_BACKLINKS_ALLOWED' for the name of whatever setting you're changing.

And that's it. Have fun.

Geeky Groups near Birmingham UK

2011-06-18T08:10:00.001-07:00

I've recently been getting more into meeting like-minded people, which is to say 'geeks'. After years of trying to pretend that I'm not one (in the same way that I tried to pretend my eyesight wasn't failing when I was a kid) I've finally decided to come out.

This new sociability probably started when I visited the c-base hackerspace in Berlin,

where I got horribly drunk and threw up in their loos. It wasn't all my fault though, I didn't pour all that Jager down my own throat. Well, okay, literally speaking I did, but someone kept filling the glass, so I feel there's shared responsiblity.

Anyways, since then I've been to the Nottinghack hackerspace in Nottingham. They did a frankly wonderful lockpicking workshop, and if they ever do another of these and you can go to it, I'd urge you to do so. You'll learn a lot (most of it alarming).

At the time I attended, they were still in the 'old' hackspace, which had two largish rooms, but was already a little crowded. They have since moved to somewhere that looks like:

It's HUGE. I should go and visit again!

Birmingham has been unlucky hackspace-wise, the hackspace there kinda folded, (Birmingham always seems to fail to do things that far smaller conurbations have no problem achieving) but now people are trying to restart it again!

FizzPop is the Birmingham hackerspace. Currently they have no permanent space and are meeting up in bars and what have you, but plans are afoot to get a permanent residence sorted.

Once upon a time, they got to this level:

Let's hope they can get back there again.

If you're a maker or hacker in B'ham, then FizzPop needs your support. They need to get some paying members so they can pay for a proper space, it's one of those chicken-and-egg things.

I guess I should say 'we' rather than 'they', because I'm kinda involved in this, but I live a ways out of Birmingham, so I'm not as involved as all that.

Finally I went to the Birmingham Linux User Group. I wasn't expecting much from this, but was very pleasantly surprised. The session I went to was just three guys standing up and doing a talk about something (one of whom was a FizzPopper that I already knew) but all three did an excellent job.

They record and post these talks, so sometime in the future the one that I attended should appear among them. If they do, you can hear me, I'm the squeaky voice asking obnoxious questions like "Why does a modern database system need something like memcached to support it?" (Imagine this said in the same manner as "What does God need with a starship?"), went down like a lead balloon that one did.

So, anyways, I'll be going to these groups again, and if you're a maker or hacker or geek in B'ham, so should you.

How to fix your broken Blogger client program

2011-06-18T00:09:00.001-07:00

For about a month now I and other people have been unable to use the blogger api to update our blogs. This is one of a raft of problems caused by a recent update of the blogger system, and some other ones (including the fact my OpenID is broken) are still outstanding. However, the OpenID thing isn't a big deal, whereas the api blog posting is something I use a lot.

I can happily report that the Blogger/Google people have come through for us and found the problem. I had been losing hope, because as a linux user I'm used to having people break stuff, and then say the problem is that I'm running linux. But, though it's admittedly taken them a while, google's blogger team have dug to the bottom of the problem and found the cause. Maybe I should have more faith in future, we'll see.

If you've found that your blogger API client program has stopped working in the last month, then what follows is what you need to do to get it working again.

Turns out that some of us had misinterpreted the blogs 'posting endpoint' to be this

http://idratherhack.blogspot.com/feeds/posts/default

which was working fine, you could send the appropriate XML to this endpoint, and post to your blog. But it was never supposed to work, and since the upgrade, it doesn't. This is a shame, because it's a nice URL that makes some sense to the user.

The endpoint we should have been using looks more like:

http://www.blogger.com/feeds/8441087078477003568/posts/default

So the question is, how do you get from a blog name/url like 'idratherhack.blogspot.com' to this horrible blog ID? Well, there's a few ways of doing it.

1) Connect to your blog and pull it as an html page. Near the top you'll see an entry like:

<link rel="service.post" type="application/atom+xml" title="I'd Rather Hack - Atom" href="http://www.blogger.com/feeds/8441087078477003568/posts/default" />
The 'href' part of this gives you the url that you should use for posting to the blog.

2) The more complex way is described in the protocol document

1: Authenticate with google using client-login or Oauth.

2: do an HTTP GET to http://www.blogger.com/feeds/default/blogs

This will return a mass of XML that gives information regarding the authenticated user. Within this are entries like:


   <link rel='self' type='application/atom+xml'     href='http://www.blogger.com/feeds/profileID/blogs/blogID' />
   <link rel='alternate' type='text/html'     href='http://blogName.blogspot.com/' />
   <link rel='http://schemas.google.com/g/2005#feed'     type='application/atom+xml'     href='http://blogName.blogspot.com/feeds/posts/default' />
   <link rel='http://schemas.google.com/g/2005#post'     type='application/atom+xml'     href='http://www.blogger.com/feeds/blogID/posts/default' />

The important one is the one that has a 'rel' value of

http://schemas.google.com/g/2005#post

the accompanying 'href' gives the endpoint you must used for posting blog entries. Here it is described as 'http://www.blogger.com/feeds/blogID/posts/default', but when you get a real feed the 'blogID' will be replaced by a big old number to produce a unique endpoint for the blog.

This XML also contains other interesting 'link' entries, like:

rel=http://schemas.google.com/blogger/2008#setting
rel=http://schemas.google.com/blogger/2008#template

the href of these gives endpoints that let you pull a list of settings or the template of the blog.

So basically, if you get this 'post' endpoint, and use that instead of http://<blogname>.blogspot.com/feeds/posts/default, you should find that your API program starts working again.

Google's Blogger Service is Broken

2011-05-27T14:33:00.000-07:00

Since about 2011/05/22 google's "Blogger" service (where this blog is hosted) has been experiencing serious problems. You can see a list of increasingly irate people here . It's a weird fault that hits different people in different ways. Some can't log in, some can't log out, some can log in but can't post.

I've not been as heavily affected as some, I can still log in to the website and post to my blog. However, I can't use the 'blogger api' to post from applications. Other people using blogging applications like MacJournal or ScribeFire are having the same problem.

The amazing thing is how long it's taking google to fix this, days. Also it seems to me (though I may be wrong) that this is linked to a recent decision by google to turn on 'Spam Checking' on everyone's blog comments. They didn't ask us if we wanted this turned on, they didn't suggest we turn it on, they just went and turned it on for us. They did this a while back with 'google buzz', turning it on and in the process revealing users gmail contacts to the whole world. I conclude from this (assuming I'm right about the cause) that google learns nothing from history.

But there's a bigger issue there. We're told that everything is moving into 'the cloud', that big competent companies will 'look after our data' and make thing effortless for us. You know, big competent companies like BP, or Toyota, or Enron, or Sony, or Google, or any bank, or...

These big companies aren't exactly covering themselves with glory these days. Would you really trust them with your business critical data?

Things I wish people had told me about 1: Bind Mounts

2011-05-10T11:01:00.001-07:00

Stuff I wish people had told me about No. 1

Bind Mounts

Quite often I find myself wanting to run processes in a chroot. This is especially true for file services being served up by ftp or http. Unfortunately I often find that files I want people to get access to are outside the directory that I want to chroot them into.

Now, symbolic links won't work across chroot, and though you can use a hard-link to make files individually available within a chroot, it becomes a lot of work for a lot of files.

Just recently though, I discovered that you can do this:

mkdir /home/PublicData
mount --bind /home/PublicData /home/chroot/PublicData

These 'bind mounts' allow you to 'mount' a directory onto another directory. So, you create an empty directory with a particular name, then call the mount command to mount another directory onto your empty directory, so that when people enter the 'empty' directory, they are teleported to the other directory.

This means that you can choose which directories to make available to people who are in some kind of chroot jail. Very useful!

I wish someone had told me about this earlier.

Root's bash profile not working when su

2011-04-12T03:38:00.001-07:00

A reaccuring annoyance for me is getting root's bash_profile to work, and everytime I fix it, I promptly forget what I did, so here's a post for me to refer to if noone else!

Things like the users PATH and other environment variables can be set up in a file in their home directory called .bash_profile. If they're using the Bourne Again Shell then this file gets read when they login, and any commands in it are run to set up their environment. But in my experience it never works for root.

There's a difference between root and other users though, I almost never login as root, I login as another user and then 'su' to root. Turns out that an 'su' doesn't count as a login, and '.bash_profile' only gets read and run for login shells. If other types of shell get started up, say by being launched from a program or whatever, then another file '.bashrc' is read instead. So, if I put the same commands in /root/.bashrc that I normally would in /home/user/.profile, then they will work when I 'su'.

Finally there are 'global' bashrc and profile files that can be put in /etc, and these will be run for all users before any profile or bashrc in their home directory is run. Note that these files don't have the leading '.', so instead of /home/user/.bashrc, it's /etc/bashrc. And for the profile, it's /etc/profile instead of /etc/bash_profile as you might expect! Oh, and finally, these global files don't seem to work for /root.

So, to get the PATH and prompt etc set for everyone, create /etc/profile for all users but root, and then copy it to /root/.bashrc, and /root/.bash_profile and that should cover 'su' and 'login' shells for root too.

Nvidia drivers fail without 'smp_lock.h'

2011-04-04T06:48:00.001-07:00

If you install a very recent kernel from www.kernel.org, and then try to install the NVIDIA graphics drivers, you may find that the driver install fails. The NVIDIA installer actually compiles the drivers against your kernel source tree, but recent kernels seem to have dropped a file that's called 'smp_lock.h'. Unable to find this file that it expects to have, the NVIDIA driver compile process fails.

A quick and dirty solution that worked for me was to simply locate 'smp_lock.h either under /usr/include/linux, or in an old kernel source tree, and drop it into the include/linux directory of your new kernel source tree. If you can't get a copy of a the file, then try making an empty file with the same name, I think even that will probably work. You should then find that the NVIDIA install process gets by that hurdle at least. It worked for me!

I'm up on "Daily Science Fiction"

2011-02-07T04:28:00.000-08:00

Well, on Friday my story "Imaginary Enemies" went out on the Daily Science Fiction mailing list. My first pro-paying publication. What feed-back I've seen from readers is good, no one hated it, many people are saying they liked it. I'd like to thank my friends and family, and Sainsbury's for doing the catering. I'm sure my sudden success won't change me.

Actually, I was kinda hoping it would change me, but no signs yet.

Anyways, in a week or so it's apparently going to get posted to the Daily SF website for everyone to see, even those who aren't on the mailing list, and no doubt then the bidding war for film rights will start!!

Broadband speedtests that don't need flash or java

2011-01-28T04:14:00.000-08:00

Look, I don't want to install flash or java just to find out how fast my internet connection is. I just want something that downloads a file and gives me a result. Here are some websites that do this basic thing.
1) Tracert.org
http://www.tracert.org/bandwidth_meter/

For me the best one, almost magical in fact. You just click 'start test', and it works. Works in old versions of firefox without flash, and seems to produce results that agree with other tests.

2) Ten Meg
http://tenmeg.myby.co.uk/

Yes, it requires you to watch a couple of images downloading and click a button when they're done, but it works in browsers without flash, java, activex, siverlight, perl, ruby, php, gtk, etc, etc, etc, etc.

3) ISP GEEKS
http://www.ispgeeks.com/wild/modules.php?name=Bandwidth_Meter_DSL

Described as their 'Mobile Speed Test', doesn't seem very accurate (perhaps because they're in the USA and I'm in the UK), but it gives a rough idea and doesn't need flash etc.

4) Broadband speed checker
http://www.broadbandspeedchecker.co.uk/

Seems to work in I.E. without flash, though it says it would like to have flash 8. Unfortunately seems to be broken in earlier versions of firefox.

5) Toast.net
http://performance.toast.net/

Again, not massively accurate, but simple and kinda fun waiting for the picture to download!

6) Mobile test
http://i.dslr.net/iphone_speedtest.html

Aimed at mobile phone browsers, but if you click the 'wifi' option, it seems to give a fairly accurate result (but that could be a co-incidence).

ALSA drivers not working for Cirrus Logic CS46xx soundcard

2010-08-25T02:06:00.000-07:00

If you are using OSS emulation for ALSA on a laptop with a CS46xx card, and are getting no sound, then read on.

I use a really old laptop (Ascentia M-Series dating from 1998) that I carry about in places where I think anything nicer would get stolen/drowned in spilled tea/rolled over by a forklift. One thing that has long annoyed me about this laptop though, is that the sound doesn't work. Things would appear to be working, mplayer would claim it was playing .mp3 files, but no sound would come out.

The sound card is a cirrus logic CS4624B, and everyone says that these are a pig to get working, requiring lots of meddling with old-style Plug-and-Pray settings because, in these old laptops, the soundcard is on an isa bus.

Funny thing is though, that back in the day when I used OSS for sound, rather than ALSA, the soundcard worked fine. Why would ALSA suddenly have irq or resource issues that require me to mess with isapnp, when OSS didn't?

In the end, the problem turned out to have nothing to do with irq's, io-ports or plug-and-play. Basically there is a bug in ALSA. For these cards, ALSA sets everything to 'Mute' by default. This doesn't mean it sets the sound level to zero, it seems to be more than that, it actually does some magic 'mute' thing to the card that you have to unlock. Unfortunately, the OSS compatibility module that makes the sound-card visible to programs that use the old OSS interface, doesn't know how to mute/unmute, so the card stays muted no matter what you do.
Or it might not be the mute, it might be the fact that one of the volume controls 'Master' doesn't appear to be exported to the OSS emulation, under ALSA volume control programs you see the 'Master' setting, but under OSS volume controls, you don't.

The only solution I've found is to install the alsa libraries, and use 'alsamixer' to initially set up the card and volume levels. Suddenly BINGO! It works! No need to meddle with irq's and isapnp.conf.

However, I didn't really want to install the ALSA libraries, that's why I'm using OSS emulation. But, more and more these days Linux is becoming another O.S. of you-will-do-as-we-say. IF you want to use software X, you will install lots of libraries and now you're going to have to upgrade your computer. If I have to upgrade my computer, then why not just put windows on it or get a mac and be done with it, hmmm??

M. T. Urself

2010-06-02T15:36:00.001-07:00

So, I had a server that was misbehaving in a truely weird way. I was accessing this server via and ADSL line, connecting into it using Secure Shell. Any large amount of data pulled from it caused the network connection to drop. But I could push as much data to it as I liked. Huge uploads worked fine, but downloads would get me disconnected instantly. Logging in and typing 'ps ax' would normally only get me half the page of output before the connection froze up. Also, anything that posted information to websites, like logging into googlemail for instance, seemed to freeze as well.

I tried altering the firewall, the router, the routing table on the server, recompiling the kernel, etc, etc, all to no avail. After each new attempt the problem would seem to go away, cruely making me think I'd fixed it, before returning again.

Now, often in the past when there's been network problems, some old unix-hacker with open-toed sandals and a grateful-dead t-shirt has been around to say; "You know man, could it be, like, the MTU?"

The MTU. That thing they had on modems, remember? No, I don't suppose you would.

The proper reply to this statement is, of course: "Get real grandad! It's the effin' nineteen-nineties, innit? 'MTU', ha! You iz well outta touch, m8."

And then, sometime in the noughties, he just wasn't there any more. I miss that dude now.

PARTICULARLY AS, THIS TIME, IT WAS THE ****ING MTU! TWO DAYS OF BANGING MY HEAD AGAINST A BRICK FIREWALL WITH MY BOSS RIDING MY ASS SAYING THINGS LIKE "Haven't you fixed it yet? I thought you were good at this stuff?" AND IT TURNS OUT TO BE THE GODDAMN LEFT-OVER-FROM-THE-DARK-AGES-OF-ARPANET SMALL-PRINT-OF-NETWORKING MTU!!

So, being a civic-minded soul I'm posting the information here, so others don't have to suffer as I did.

The MTU is the "Maximum Transmission Unit". It's a value that tells the computer what the maximum size of network packet is that can be transmitted on a given network. There's a "Maximum Receive Unit" too, but this is less often messed with. The MRU is the maximum packet size that a network device will accept. Obviously, people transmitting to you have to be using an MTU smaller than your MRU, otherwise you'll reject their packets as 'too big'.
The reason you can't transmit packets (sometimes called 'frames') of any old size is that most networks can only actually carry one chunk of data at a time. They achieve the 'magic' of many people using the network at once by a simple method of breaking messages up into packets, and letting everyone take it in turns to send one packet. If the packets and small, and the network moves them fast, it seems like everyone is using it at the same time. But if one person could make a huge packet, I mean really humungous, the mother of all packets, then that single packet could clog up the network and everyone would have to wait for it to have finished transmitting, before they could get a slice of the action. The slower the network, the smaller the packets have to be to provide the illusion of multi-user networking.
Another issue with mtu size, is noisy network connections. If you have networks where there is a lot of noise, and packets are likely to get corrupted, then the bigger the packet, the more likely it is to be 'hit' by corruption. This will mean that the packet has to be re-sent, but because it's a big packet, it's quite likely to get hit by corruption again!

Modems and serial links were generally noisy links, and they were slow too. As a result they generally had small MTU sizes, and much time was spent tweaking the MTU size to get the best results on a particular line. However, ADSL/Cable and Ethernet are much faster and more reliable, and generally all seem to agree on a default MTU of 1500 bytes.

There's also some magic in TCP/IP called 'Path MTU discovery', which works like this: If there's a network device between you and the machine you are exchanging data with, that cannot handle your chosen MTU/packet size, it will drop your packet and send a message back to you saying 'packet too big'. This message is sent using the icmp protocol (the same protocol that powers 'ping'). Hence, most modern systems should automagically reset their MTU to whatever is needed.

However, if you block icmp with your firewall, because you're thinking "I don't need ping", then this magic won't be able to happen. If you then also have an ADSL router who is using an MRU smaller than your MTU, it will drop any large packets you send it, and your system won't know because it's icmp based complaints won't get through your firewall. Hence, you'll be able to connect to your computer from outside, or connect to websites and pull data from them, but the first time you pull more than a little data from your machine, or push more than a little data to the net, everything will lock up.

The Cure

There's two things to do:

1) Find out what MTU/MRU the device you are talking through (in my case ADSL router) is using, and use /sbin/ifconfig to set the network card to use it.

/sbin/ifconfig eth0 mtu 1496

2) Make sure you let 'icmp' through you firewall so your system can adjust the MTU automagically.

/sbin/iptables -I INPUT -i eth0 -p icmp -j ACCEPT
/sbin/iptables -I OUTPUT -o eth0 -p icmp -j ACCEPT

Of course, doing item '2' allows people to 'ping' your machine. There are ways to prevent that with iptables, but these are left as an exercise for the reader.

Quick and Dirty Webcam monitoring

2010-04-16T14:22:00.001-07:00

Just thought I'd mention a very quick and dirty way to get 'webcam monitoring' going. I've done this with axis webcams, but it should work with any webcam that allows you to fetch a jpeg image from a url.

This will, of course, be working under linux. It's only using the standard 'Borne' shell and a program called 'links' so it should work under FreeBSD and Mac OSX too.

The command-line webbrowser, 'links' has an option called -'-source'. This has the effect that a webpage or item fetched using http with links is written in raw format to stdout. So

links -source http://media.libsyn.com/media/djsteveboy/the_swing_set.mp3 > TheSwingSet.mp3

Will download the example .mp3 file.

Most axis webcams allow you to download an image from http://<webcamip>/axis-cgi/jpg/image.cgi, so we can use links to Grab an image thusly:

links -source http://192.168.5.4:1000/axis-cgi/jpg/image.cgi > snapshot.jpg

Of course, what's best is to get images every couple of seconds and put them into a date/timestamped file. Here's the script I've been using, it creates directories for each hour of a day, and puts timestamped .jpg files in them.

#! /bin/sh

cd /home/Public/AxisWebcam

while [ 1 = 1 ]
do

DIR=`date "+%d_%m_%Y_Hour-%H"`
FILE=`date "+%H_%M_%S.jpg"`

mkdir $DIR
links -source http://192.168.5.4:1000/axis-cgi/jpg/image.cgi > $DIR/$FILE

sleep 2

done

One issue is that links doesn't have any way (that I know of) of expressing a username/password to access a website. Fortunately, Axis cameras have a setup option under 'Setup->Users' called 'Enable anonymous viewer login'. This allows access to the image without having to supply a username and password.

If you're going to leave this script running, you may want to periodically run a command like:

find -mtime +7 -exec rm -f {} \;

This command will delete any files older than a week.

And that's it!

Slow boot due to libata problems with mwdma2

2010-03-09T14:04:00.001-08:00

I use a lot of old kit. Recent linux kernels are starting to have problems with old kit. An old laptop harddrive suddenly started to 'freeze' during bootup. The kernel was trying to access it with various DMA modes, none of which seemed to work. Finally it would fall back to PIO, and work, but this was annoying, because it meant the laptop took ages to boot.

I'm attaching a dump of what the kernel spewed out while it was trying to talk to the hard-drive. The main thing to look for if you have the same problem is:

[ 23.107017] ata2: soft resetting link
[ 23.259250] ata2.00: configured for MWDMA1

and it will gradually work thorugh the possiblities until

[ 65.107128] ata2: soft resetting link
[ 65.262031] ata2.00: configured for PIO4

What you want to do, of course, is tell the kernel 'just go for PIO4 straight away', and at least it will boot quickly.

Turns out you can!

I added 'libata.force=2:pio4' to my kernel 'append' line. I use lilo to boot my system, so it's as simple as:

append="libata.force=2:pio4"

added to my lilo.conf

This tells the bit of the kernel that deals with harddrives (libata) to use pio4 (not sure what that is, but it works, whereas mwdma2, which I'm also unsure what it is, doesn't) on the second 'ata' interface.

I gathered that it was the second ata interface from all the bootup messages about 'ata2'

And it works fine, if, I suspect, rather slowly.

So if you're seeing the following in your logs, you probably want to do this.

[ 1.671100] scsi 2:0:0:0: Direct-Access ATA TOSHIBA MK1403MA D3.0 PQ: 0 ANSI: 5
[ 1.677462] sd 2:0:0:0: [sda] 2818368 512-byte logical blocks: (1.44 GB/1.34 GiB)
[ 1.678301] sd 2:0:0:0: Attached scsi generic sg0 type 0
[ 1.682247] sd 2:0:0:0: [sda] Write Protect is off
[ 1.682439] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 1.684011] sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1.687772] sda: sda1 sda4
[ 1.808661] sd 2:0:0:0: [sda] Attached SCSI disk
[ 23.105610] ata2: lost interrupt (Status 0x5)
[ 23.105777] ata2: drained 4 bytes to clear DRQ.
[ 23.105848] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 23.105989] scsi 3:0:0:0: CDB: Inquiry: 12 00 00 00 60 00
[ 23.106497] ata2.00: cmd a0/01:00:00:60:00/00:00:00:00:00/a0 tag 0 dma 96 in
[ 23.106520] res 40/00:02:00:24:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[ 23.106841] ata2.00: status: { DRDY }
[ 23.107017] ata2: soft resetting link
[ 23.259250] ata2.00: configured for MWDMA1
[ 23.259684] ata2: EH complete
[ 44.105592] ata2: lost interrupt (Status 0x5)
[ 44.105752] ata2: drained 4 bytes to clear DRQ.
[ 44.105815] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 44.105954] scsi 3:0:0:0: CDB: Inquiry: 12 00 00 00 60 00
[ 44.106456] ata2.00: cmd a0/01:00:00:60:00/00:00:00:00:00/a0 tag 0 dma 96 in
[ 44.106480] res 40/00:02:00:24:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[ 44.106798] ata2.00: status: { DRDY }
[ 44.106970] ata2: soft resetting link
[ 44.259253] ata2.00: configured for MWDMA1
[ 44.259674] ata2: EH complete
[ 65.105597] ata2: lost interrupt (Status 0x5)
[ 65.105758] ata2: drained 4 bytes to clear DRQ.
[ 65.105819] ata2.00: limiting speed to PIO4
[ 65.105946] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 65.106086] scsi 3:0:0:0: CDB: Inquiry: 12 00 00 00 60 00
[ 65.106655] ata2.00: cmd a0/01:00:00:60:00/00:00:00:00:00/a0 tag 0 dma 96 in
[ 65.106679] res 40/00:02:00:24:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[ 65.106954] ata2.00: status: { DRDY }
[ 65.107128] ata2: soft resetting link
[ 65.262031] ata2.00: configured for PIO4
[ 65.262467] ata2: EH complete

Linux Tips: Vim autoconfigure for different filetypes

2009-12-17T08:52:00.001-08:00

Thanks to Ajay Somani, over at http://ajayfromiiit.wordpress.com/ for this post that finally helped me solve a long term gripe with vim.

When I'm in 'programmer' mode, I use vim with syntax highlight turned on.
When I'm in 'wannabe fiction writer' mode, I use it with spell check turned on.

And these two personas keep fighting like Jekyll and Hyde. Because I have both of these turned on, vim insists on syntax highlighting my fiction, interpreting apostrophes as quotes and then coloring everything after them. Worse still, it spell-checks my C-code. It'll highlight some text in red, and then spell-check it, and decide it's mis-spelt, and thus surround it in a background colored.... red. This results in large blocks of red-on-red text that cannot be... uh... read.

The solution, it turns out, is to add this magic line to my vimrc:

autocmd BufEnter *.txt set spell

Basically autocmd is a way of specifying a vim command...

I don't believe this, vim is coloring every instance of its own name in purple as I type this. I'm pretty sure it's doing this out of pure vanity.

... anyways autocmd specifies commands to be run when an event happens. the 'BufEnter' event, though it doesn't sound like it has anything to do with files (and indeed, sounds vaguely rude) gets run whenever you start editing a new document (which by and large, means a file). It takes a command which is a pattern to match a file name. Then you supply the command, in my example 'set spell', which only turns spelling on for .txt files. So:

autocmd BufEnter <file pattern> <command>

Another useful one, I find, is:

autocmd BufEnter *.xml set nosyntax

Because I'm often confronted by .xml documents that have no line breaks, and are just one HUGE line. This really upsets vim if you have syntax highlighting turned on, because I think it normally does syntax hilighting a line at a time, not bothering with those lines that aren't being displayed. But when faced with an xml document that's just a single giant line, it has to highlight the whole document, and the user can go off and make a cup of tea while it does so.

I'm sure there are lots of other uses for the various events that autocmd supports, and if anyone knows any, I encourage them to comment here!

HTTPS: No one uses it

2009-12-16T01:27:00.001-08:00

I spent a whole night battling to get secure http working in 'Movgrab', only to discover that no-one uses it. I couldn't find a single proxyserver that supported 'straight' https (some of them might support use of the 'upgrade' header, but as of right now, I don't understand that). So then I switched to using https to connect directly to video-sharing websites. Any google website (youtube, videos.google.com) bounces me straight to www.google.com if I try to use https. Many other sites let you use it for the initial connection, but insist on redirecting you to http: when you try to get video. The rest don't support https at all.

I don't know, I would have thought that in these days of 'Web 2.0' secure browsing would have been the norm by now.

Hey, movgrab people!

2009-12-15T02:19:00.001-08:00

So, I posted movgrab to www.freshmeat.net on Sunday. Since then, I've not heard anything, but through my advanced psychic superpowers I happen to know that visits to my new technical blog have jumped dramatically. The only explanation for this, I believe, is that people are downloading movgrab, and seeing the mention of my blogs on the help-page (or the website they download it from).

So, if you're coming here having used movgrab, leave a comment. Did it compile on your system? Does it work as advertised? Will it change your life? Did using it open a portal to the dungeon dimensions (if it did, and you're thinking of suing, you're out of luck, I included a disclaimer covering exactly this eventuality).

Come on people, I needs to know!

My first code release up on freshmeat!

2009-12-13T20:52:00.000-08:00

Big day for me today. I've been writing software projects for simply years, maybe even decades, and every one of them has grown and grown, but it's never finished, it's never ready for release. Finally I've gotten my first project completed!

Okay, it's a fairly trival app. Movgrab is a command-line app for downloading flash/mp4/mp3 movies and audio from websites that offer that kind of thing. This is useful to we linux luddites who are still using machines that most people wouldn't even use for email these days (for instance, I've got a bunch of pentium-1 laptops that I use as 'holiday' or 'travel' laptops, because no one is likely to bother stealing them, and if they get smashed or something, then I've not lost very much). On weedy machines like the one I'm typing this blog-post on, web-browsers like firefox don't run well. Unfortunately light-weight webbrowsers don't support flash. The solution? Download the movies and watch them with a command line player, rather than in a web-browser.

There are other downloader applications out there of course, but few of them are command-line 'C' programs like movgrab. Most of them require you to install scripting languages and libraries, which can be a headache, I find. Also, few of them support as many download types as movgrab does (including BBC iplayer, youtube, metacafe, dailymotion, crazymotion, uchannel, various university webcasts etc, etc).

More important than the app is the underlying library ('libUseful') that allowed it to be written. Writing movgrab meant adding a lot of code to the library to support HTTP transfers, and this is being used in a lot of other projets too.

So, there is a new website over at http://sites.google.com/site/columscode, where one can download this first of my masterpieces.

Linux tips: Recovering lost text data from disk, or EVEN MEMORY

2009-11-29T10:18:00.000-08:00

Okay, first new post for the new 'technical' blog. The topic is "Holy ****!!! I just had my browser/text editor crash, and I've lost all that work! What do I do now?"

Well, so long as the editor you were using stores it's data in some form of plain text, and so long as you are using linux, freebsd, or some other unix, (or possibly Mac OSX), I may be able to help.

WARNING: These techniques, like any secret powers, come with a degree of risk. I take no responsibility for any accidents where you wipe your hard-disk, explode your computer, or accidentally sell your soul to the devil. In fact I take no responsiblity for anything at all. You have been warned.

I use opera as a web-browser, and every now and then it crashes. This often happens when I'm in the middle of typing a big old 'mr angry' post onto a web-forum somewhere. Other times, I'm writing code, or SF stories (I am, like so many people, a wannabe writer) and a muscle spasm causes me to kick the power-supply of my computer, disconnecting it. Or there's a power cut. Or more often than this it's the shameful FAIL!! of doing 'rm -r -f' in the wrong place.

Fortunately, I've always (so far) been able to recover my work when such things happen. The tool I use for doing this is 'grep'.

Now, as you probably know, 'grep', whose name stands for 'Get Regular ExPression' is a tool that prints out lines matching a certain pattern in a file. But, if you had a file, you wouldn't have lost the work, would you? Well, lets consider our old freind, the ill reckless 'rm -r -f'. You thought you were in the /tmp directory, but actually you were in the /home/MyLifesWork directory, and now you've wiped all that oh-so-important stuff. You do have backups of course, don't you?

Well, if you don't, you may be able to at least recover any plain-text documents. When you 'delete' a file, what generally happens is that the chunk of hard-disk that the file was on gets marked as being free for re-use. But it doesn't get 'wiped clean', the data is still in there. So long as you can find this chunk of hard-disk and read the data out of it before it gets used again, you can recover much, maybe all, of your work.

You'll need to be 'root', i.e. the superuser on the unix box you are trying to recover from. As ever, mistakes made as 'root' can be disasterous, so be sure you know what you are doing. (Me, I'm always logged in as 'root'. Yes, I know, that's really bad practice. Do as I say, not as I do).

Linux has a directory called '/dev'. In this directory are 'virtual files'. These are not files in the sense we would normally understand them, but rather the input-output endpoints of device drivers. Devices like your hard-disk and computer memory appear in /dev as though they were files. You can read and write to the entire hard-disk, as though it were one giant file. I really wouldn't recommend writing anything to these files, as this is a good way to wipe your hard-disk clean!! BE VERY SURE YOU KNOW WHAT YOU ARE DOING HERE.

So, depending on what kind of hard-drive you have, the device files will probably be called something like '/dev/hda' or '/dev/sda' for the first hard-drive, and '/dev/hdb' etc for the second. In addition to this, there will be device files for the partitions on those hard-drives, named like '/dev/hda1', '/dev/hda2' etc.

If you know a word or phrase that occurs in your lost document, you can do

grep "My phrase" -a /dev/hda2

to grep for it in the entirity of the 2nd partition on the 1st hard-drive. Further, you can redirect this output to a file like:

grep "My phrase" -a /dev/hda2 > /tmp/MyPhrase.save

However, there is a danger here. The 'lost file' takes up a chunk of hard-disk that is available for re-use, so the second that you start writing to '/tmp/MyPhrase.save', you run the risk of overwriting the very data you are hoping to recover! It's better, if you have another partition you can use, to copy the data onto that partition, not the partition that you are trying to recover data from. For example, you might have a usb-disk or pendrive that you can mount on /mnt, and then redirect to '/mnt/MyPhrase.save' rather to '/tmp/MyPhrase.save'. This will mean that you aren't writing onto the same hard-disk you are trying to recover from. If it's a usb-disk, this might be slow, but better slow than overwritten!

Well, this is all very well, but grep only returns lines that contain the searched for phrase. You want the whole document! Well, grep takes two command-line arguments:

-A Return after matching line
-B Return before matching line

For instance, half the reason that I'm writing this post right now, is that I lost my 'Margaret Atwood: Bad girl, or just Misunderstood?' post that I was writing for my other 'non technical' blog in a freak accident. I was able to recover it by grepping for 'Misunderstood'. I knew that 'Misunderstood' would be at the start of the document, so I only had to use '-A', like this

grep -a Misunderstood -A 200 > /tmp/MA.save

This will cause grep to print out every line it finds with 'Misunderstood' in it, and also the 200 lines that follow that one.

If I'd known that 'Misunderstood' was at the end of the document, then I'd have used -B. If my search phrase is mid-way through, then I'd use -A and -B in combination to grep lines before and after the phrase.

Okay, so we have our file of saved data. Unfortunately, that's only the beginning. Looking in the file with a text-editor (I recomend 'vi') you will find that, frankly, it's full of crap. Oh, your missing work is in there somewhere, but there's loads of extra stuff been pulled along with it. You'll have to go through and pluck your work from the mess. Sorry, that's how it is.
Also, if your missing document is fairly large, it's unlikey that it will all have been stored in one place on disk. So, I grepped 'Misunderstood', and that got the the first third or so of my blog-post. To get the rest, I had to grep for other words. grepping 'Atwood' returned all kinds of chunks of the document, and in the end I was able to sew these back together into my original text. It's a messy proceedure, but in the end, it works.

One tool that can help you with this is 'strings'. 'Strings' takes a file, and only prints out the text in it that uses a restricted set of human-readable characters. Most of the 'control characters' and binary stuff, it throws away. So one could either do:

 
grep -a Misunderstood -A 200 > /tmp/MA.save
cat /tmp/MA.save | strings > /tmp/MA.strings

 
grep -a Misunderstood -A 200 | strings > /tmp/MA.save

And this will eliminate some of the mess that you have to consider.

But here's something else about my lost 'Margaret Atwood' post. It wasn't saved on disk at all, I was typing it into a text box in my opera browser when, for some reason best known to itself, opera went 'plink'. How can you recover something that's not even written to disk, an only existed in the memory being used by a given application?

Well, it depends if your unix operating system blanks all memory down when it is freed. Some 'ultra secure' versions of unix do this (you can patch the linux kernel to do this) to ensure that someone naughty, who is logged in as root, can't use these techniques to look into memory you are using, and see all your dirty little secrets. However, if you don't have such an 'ultra secure' unix, you may be in with a chance. Indeed, about an hour ago, I managed to recover my 'Margaret Atwood' blogpost from memory.

First off you should probably shut down as many programs as you can, to prevent them from grabbing the memory that your 'crashed' application has just given up. WHATEVER YOU DO, DO NOT TURN OFF OR REBOOT YOUR COMPUTER, as this will obviously blank all memory.
Then, if you look in /dev, you'll see a file /dev/mem. This file lets you view your memory as a file, just as you can view hard-drive partitions as a file, so all the same proceedures can be used to recover something that's still hanging around in memory.

The results of grepping /dev/mem tend to be even messier than grepping hard-drive partitions, but your work should be in there somewhere. You'll have to find the chunks and join them together.

If you were editing an html document, you might find that it's full of things like '%20' indicating a space. You can replace these using 'sed'

cat /tmp/MA.save | sed "s/%20/ /g" > /tmp/MA.cleaned

similarly for any other characters that have been 'quoted' using the http quote method. Characters like '/' and '"' and '[' have special meanings to sed, so you'll have to quote them with '\' in your sed command, so that sed knows what you mean. For instace:

cat /tmp/MA.save | sed "s/%2F/\//g" > /tmp/MA.slashes

if you wrote '/' instead of '\/', sed will get confused, because it uses '/' to divide 'thing to replace' and 'thing to replace it with'.

So, that's all you need to recover text, html, and probably a few other 'text based' formats.

Good luck. You'll probably need a little luck.

I christen this blog...

2009-11-29T09:58:00.000-08:00

So, I'm starting up a second blog. Why? Well, my first blog was about my efforts to be a writer, my sundry opinions on the state of Science Fiction and the political issues within it, and technical linux hackery. These things did not play well together. People who knew me from my 'writerly' efforts were completely bamboozled by posts full of 'grep | cut -c 1-10 | sed "s/apple/pear/" | tr ':' ' ''

And can you blame them?

But, on the other hand, my rare 'technical' posts got the most satisfying feedback, when people said "Hey! This really helped me to do something!", so I'm starting up this 'Linux techy' blog to turn alongside my main blog over at thesingularitysucks.blogspot.com

First I'll be transfering over my old 'linux' posts, and then I'll come up with something new to say.

By-Voice: Free online text-to-speech synthesis

2009-06-20T08:48:00.000-07:00

I stumbled across a cool little service at by-voice.com. Basically, this is a simple web-service by which you can get text converted to computer-generated speech. The service works by making a HTTP request to by-voice, which means its a little awkward for everyday users, but it's wonderful for programmers, because you could, if you were to sign up to the service fully, write programs that would on-the-fly download speech and play it.

The free service gives you 200 speech-generation credits a month. But unfortunately this doesn't mean 200 downloads, as we shall see later.

There are a bunch of different 'speakers':

carlos - Spanish (European) Male Voice
laura - Spanish (European) Female Voice
amaya - Spanish (European) Female Voice
jennifer - English (US) Female Voice
daniel - English (UK) Male Voice
jane - English (US) Female Voice
celia - Spanish (Mexican) Female Voice
oriol - Catalan Male Voice
meritxell - Catalan Female Voice
empar - Valencian Female Voice
amaia - Basque Female Voice
adriana - Portuguese (European) Female Voice
julia - Portuguese (brazilian) Female Voice
brigitte - French Female Voice
freire - Galician Male Voice
javier - Spanish (Argentina) Male Voice
isabel - Spanish Female Voice

Unfortunately they can only read text in their own languages, which is a shame because I'd quite like to have my laptop speak to me in exotic female voices.

Anyways, to use the service you have to sign up and get sent a 'developer key' in your email. Then you can download voices by crafting a web-url like this:

http://www.by-voice.com/ttsonline/get_speak_3scale.php?account=your3scalekey&voice=carlos&format=mp3&text=hola%20mundo

This one will make 'Carlos' say 'Hola mundo!', which I suspect is "Hello World" in spanish. Notice that you have to change 'account=your3scalekey' to be whatever the key you got in the email was. Furthermore, note that all spaces in the speech-text have to be replaced by '%20', which is the standard HTTP quote mechanism.

Obviously, you can change the speaker by changing the 'voice=' argument. So, 'voice=jane' gets you speech in a female US accent. If you send her "Hola mundo" though, she'll probably make a terrible mess of it, best to make her say "Hello World".

'format=' can be changed to 'format=wav' if you want microsoft .wav files. However, they cost more credits. Yes, I'm afraid that you can't just keep downloading as many phrases as you like, the free sign-up only gets you 200 credits. 200 is a lot, right? Well, no, a credit is PER CHARACTER, so you can quickly use up all your free credits, and have to wait a month before you get another 200. For some strange reason each character spoken in .mp3 format costs only 1 credit. The same character in .wav costs 4 credits.

Still, you can have lots of fun with this, and I'm even thinking of signing up to the 45 euro paying plan for a month, which gives 45000 credits. To be honest though, I think by-voice would do better to have a web-frontend where you could by just the credits you need, when you want them.

Okay, here's how to use the service under Linux. There are a bunch of command-line programs one can use under linux that will copy the contents of a web-url to disk, wget, curl, etc etc. I use 'links -source', which uses the links webbrowser as a downloader rather than a browser. So:

links -source "http://www.by-voice.com/ttsonline/get_speak_3scale.php?account=your3scalekey&voice=carlos&format=mp3&text=hola%20mundo" > hola_mundo.mp3

Will get the file for me. Notice that I had to put the whole url in quotes, this is because '&' is a special character for the shell, and will confuse things if you don't quote the string.

Obviously, typing in that url all the time is a pain, so I wrote a trivial little script, 'byvoice.sh':

VOICE=jennifer
KEY=<insert your developer key here>
FORMAT=mp3

TEXT=`echo $* | sed "s/ /\%20/g"`

links -source "http://www.by-voice.com/ttsonline/get_speak_3scale.php?account=$KEY&voice=$VOICE&format=$FORMAT&text=$TEXT"

This does all the work like replacing spaces in the text, so now I can just type
./byvoice.sh "Battery low" > BatteryLow.mp3

And I have my .mp3. Sometimes you have to play about a little with the text to get them to say it right. For instance, wanted speech to tell me when Dan Carlin updates his 'hardcore history' podcast. I started with:

./byvoice.sh "New hardcore history podcast available" > NewHardcoreHistory.mp3

Unfortunately 'Jane' isn't familiar with the word 'podcast', and pronounces it 'powwdcast'. I solved this by breaking it into two words 'pod cast'. Then she gets it right.
She does even worse with 'Hardcore', managing to pronounce it to sound like 'Hard porn', which is not something that I want to have coming out of my laptop when others are around (hence, all the 'hardporn podcast' alerts are silent!). The solution here was to replace the 'c' in 'core' with 'k'. So finally:

./byvoice.sh "New hardkore history pod cast available" > NewHardcoreHistory.mp3

And I have what I want. But then I'd used up all my credits.

So, there it is. Go over to 'www.by-voice.com' to play with it, although start off with SHORT sentences if you want to get anywhere with your 200 credits.

Linux Tips: SSH

2009-06-07T06:13:00.000-07:00

It's been a while since I posted, having gotten most of my rants off my chest in the early days of the blog. Rather than just spout my opinions again, I thought I'd do some posts about Unix/Linux tricks that I find myself using time and again.

First up, uses of secure shell 'SSH'. These tricks should work with any unix that has an ssh command, so they should also work with MacOSX, I think. Obviously, you need two machines for them to communicate between, or it's all pointless.

Ssh is a rather wonderful tool, it's like telnet but encrypted. (If you don't know what telnet is, then it's probably not for you.)

SSH isn't just able to work as a remote terminal, you can pipe the output of a command into it, and send that to a remote machine. Or you can run the command on the remote machine, and have it's output come spewing out of SSH.

Run a command on a remote machine (in this case the date command), and see it's output:

ssh user@remote_machine.com "date"

Run a command on a remote machine and pipe it's output to a file:

ssh user@remote_machine.com "cat /var/log/messages" > messages.remote_machine

Run a command on a remote machine and pipe it's output to a local command (this should get the recent diagnostic messages from the remote machine, and pipe them into a mail command on the local machine).

ssh user@remote_machine.com "dmesg" | mail colum

Run a command on the local machine, and send it's output to the remote (sends the diagnostic messages from the local machine, and counts the lines in it on the remote. I know it's not very useful, but it's just an example, okay?)

dmesg | ssh user@remote_machine.com "wc -l"

One thing I consistently use this for is shifting large numbers of files from one machine to another using the 'tar' command, like this:

tar -zcO file1 file2 *.dat | ssh colum@remote_machine.com "mkdir Backups; cd Backups; tar -zxf -"

if you don't know tar, it's an old-school 'zip' utility for unix. The '-zxf' is a condensed form of the command line switches '-z -x -f', as tar lets you run them together. The tar command

tar -z -c -f tarfile.tar.gz file1 file2

Makes a 'tarfile' called tarfile.tar.gz, and this is compressed using gzip compression:
-z = use gzip compression
-c = create tarfile
-f <filename>

you can unpack the tarfile with -zxf tarfile.tar.gz, where 'x' means 'extract files'.

To do our 'send to remote' trick though, we need to not make a tarfile, but to pump the tar output to standard out, which we can then pipe into our ssh command. The command line switch for this is '-O', which, if you don't pipe it into anything, will cause tar output to spew all over your screen. Logically, if 'send to standard output' is '-O', then 'read from standard input' should be '-I'. But it's not, to do this you type '-f -', and this gives us our full command:

tar -z -c -O files | ssh user@somewhere "tar -z -x -f -"

You can add extra commands to the command line at either end, provided they don't print out any output, which will confuse tar. Usually these are commands to make or change directories:

cd /home/colum/mywork ; tar -zcO files | ssh user@somewhere "cd /home/colum/mywork.backup; tar -zxf -"

if you want better compression during the send, and have the bzip command on your system, try replacing '-z' in the command line switches with '-j'. Obviously we have to do this at both ends.

If you want to 'Pull' files from the remote machine, rather than 'Push' too it, do this:

ssh user@somewhere "tar -zcO *" | tar -zxf -

Obviously '*' will back up all files in the login directory on the remote machine. You knew that, right?

Another thing I use this for, is synchonizing clocks between machines. The 'date' command will take the command-line-switch '-s <date>' which lets you set the date/time on a machine. So a short script like:

DATE=`date`
ssh root@remote_machine "date -s \"$DATE\""

Will set the date, provided that the user has access rights to do such things (which is why the login in 'root' here). Notice the horrible quoting of the $DATE argument on the remote machine. This is because we have to stop spaces in the date being interpreted as breaks between arguments, because the date is all one argument. So we need \" in order that the local machine doesn't try to interpret quote marks intended for the remote machine.

Another use, is watching for changes in files that shouldn't change on a remote machine. You can use the 'md5sum' command to get an md5 checksum for each file. An 'md5 checksum', in case you don't know, is a string of gobbledegook that is supposed to be unique to a file, i.e., no two files should have the same md5. Now, it's actually true that, as there are an infinite number of possible files and an md5 string is only so many bytes long, there *must* be an infinite number of files that have the same checksum. However, the likelyhood of your ever encountering two such files in normal use is vanishingly small. It's the same reasoning as the fact that there is a real possibility of all, or most, of the air molecules in your room choosing to move up one end of it, leaving a suffocating vacuum at the other end. However this never happens because it's tremendously unlikely, as air molecules famously can't get their shit together to be that organized. If you waited forever, you would see it happen, in fact, if you waited for an infinite amount of time, you'd see it happen infinite times. But it's utterly unlikely that you'll see it in your flickeringly brief lifetime, human.

Anyways, I routinely check the /bin /sbin /usr/bin /usr/sbin directories on linux servers, as these directories contain standard programs that should never change. If they do change, then something or someone has changed them, and this may indicate foul play.

ssh user@remote_machine "md5sum /bin/* /sbin/* /usr/bin/* /usr/sbin/* | sort" > remote-files.md5

This gives me a sorted file with md5 sums of the programs in it. Once I have this, I can get a fresh version each day, and write that to a different file, say 'remote-files.latest.md5' and then use a diff command to do a compare between the two. Something like this:

ssh user@remote_machine "md5sum /bin/* /sbin/* /usr/bin/* /usr/sbin/* | sort" > remote-files.latest.md5

VAL=`diff remote-files.md5 remote-files.latest.md5 | grep "^<" | wc -l`

if [ "$VAL" -gt 0 ]
then

echo "EEEK!! What's going on!?" | mail -s "Files changed on remote machine!" colum@somewhere.com

fi

Note that the 'diff' command produces complex output, containing, amongst other things, the lines from each file that differ. The lines from 'file 1' are prefixed with a '<' and the lines from file 2 with a '>'. This means you get two lines for each difference (at least). Grepping for lines starting with '<' means you get an accurate count of the number of differences.

Okay, getting more advanced, an easy way to do incremental backups is to use 'md5sum' to checksum files on both machines, and then use 'diff' to find differences between these. Here's the script:

cd mywork

find . -exec md5sum {} \; | sort > /tmp/mywork-local.md5sum
ssh colum@remote_machine "cd mywork ; find . -exec md5sum {} \; | sort" > tmp/mywork-remote.md5sum

diff /tmp/mywork-local.md5sum /tmp/mywork-remote.md5sum | grep '^<' | cut -d ' ' -f 4- > /tmp/mywork.diff

tar -T /tmp/mywork.diff -zcO | ssh colum@remote_machine "cd mywork; tar -zxf -"

This will build md5 checksums of the files in the directory 'mywork' on both machines, compare them for differences, and then backup those differences from one machine to another.

We have to use 'find' to run md5sum against all the files, because md5sum doesn't come with a useful 'recursive' option to say 'If there's a directory, go into that, and md5sum the files in that too'.

Then we do a diff between the two lists. Diff produces output that only shows lines that differ between the two files, and prefixes them with '<' or '>' to indicate which file the line is taken from. So, if we do:

diff /tmp/mywork-local.md5sum /tmp/mywork-remote.md5sum | grep '^<'

Then we only get the lines that say 'this is different on the local machine'. The following 'cut' command then clips out the filename from the md5sum output line, so finally we have a list of filenames. We use the '-T' option to tell tar to take the list of files to archive from a file (our diff output file) and thus we are sending only those files that are different between the two machines, to the remote machine.

Okay, the final thing. All the way through, if you've been trying any of these commands, you'll have found that SSH prompts you for a password. Obviously, if you are wanting to make scripts like these automated to do backups or what-have-you, then this is a problem. The solution is to use 'public key authentication', though this can be a bit of a flog to set up.

SSH accepts a command line switch '-i <filename>', which tells it a file that contains a public key to use to authenticate. So far, so easy, but setting a machine up to accept this always seems a little awkward. Basically:

1) as the user that is going to log into the system, run the command 'ssh-keygen'. It will ask you a bunch of stuff, just hit 'enter' to accept defaults every time.

2) You should now have, in the directory '.ssh' in the users home directory, two files. id_rsa and id_rsa.pub. Rename id_rsa.pub to be 'authorized_keys' and set it to have read-write permissions only for the file owner (so 'ls' shows: '-rw-------' )

3) you will have to make sure that the users home directory, and the .ssh home directory only have read, write and execute permissions for that user (this can be a problem with 'actual' user directories, best to create a user especially for automated processes to log in).

4) Take the other file that ssh-keygen produce ('id_rsa') and put it on the machine that you want to log in from. You can then log in with:

ssh user@remote_machine -i id_rsa

This won't ask for a password (unless something went wrong with the setup) so it's usable for automated processes running on a 'cron job' overnight.

Personally, I tend to rename the 'id_rsa' file to be 'id_rsa.name_of_remote_machine', because that way I can store a bunch of them in the same directory together, and I can also write scripts that work through machine after machine, simply by doing something like this:

for MACHINE in "machine1 machine2 machine3"
do
ssh user@$MACHINE -i id_rsa.$MACHINE "md5sum /bin/*" > $MACHINE.md5
done

Obviously, for this to work you have to rename the id_rsa files to be id_rsa.machine1 and id_rsa.machine2 etc, and you have to have something that maps 'machine1' to an ip address. I normally just put entries for them in the hosts file of the machine I'm logging in from.

Okay, I think that's enough of that. If you understood it, then I hope some of it is some use to you!

The Hacker Will Always Get Through

2009-05-04T05:41:00.000-07:00

The BBC currently has a program up on the explosion in cybercrime.

http://www.bbc.co.uk/iplayer/episode/b00jyyl0/Hacked_to_Pieces/

Initially this program had me shouting "No! No! No! You just don't get it!", but eventually it did a twist-in-the-tail and hit the real reason that cybercrime is so rife:

Computer users are stupid.

I'm sorry, I know it's not a pleasant truth, we would far rather believe that there are these fiendishly clever, evil people out there who are doing super-clever things, and that's why people get hacked. But the truth is never pleasant, is it?

The simple fact is that a lot of people who get hacked, deserve to get hacked. And I can put my hand up here and say that I once did something stupid myself. I was in a youth hostel, and wanted to connect to my home computer and read my mail. I use Secure Shell to do this, and was surprised to see that there was a secure-shell program right there on the desktop. That's a rather surprising thing, I thought it was suspicious at the time, but I clicked on it and used it. And lo and behold, someone got into my default account on my computer (fortunately there wasn't much there that they could do, as they lacked the higher-level passwords needed to switch to other accounts). But still, I was stupid, and things followed from that. I'd been stupid, and got hacked, and deserved it.

However, in most programs I see that address these issues, there is often a focus on the infrastructure that criminals use to organize themselves. Take IRC. When people here that criminals generally use IRC as their meeting place, there's often crimes of "Why isn't that taken down? *I* don't use IRC, clearly it is only used by criminals! Remove it and the criminal problem will go away." No, it won't. The criminals will just move onto another communication method. This is equivalent to living in a world in which everyone leaves their front doors open, but the popular solution to the resulting crime is that the roads are too open. The roads allow criminals to move easily from place to place, and make good their escape, and that what we need to do is put roadblocks and checkpoints everywhere, or even dig the roads up!

The problem that leads to cybercrime is that most people leave their front doors open, and in fact, we've been leaving our front doors open since the dawn of consumer computing:

1) Most computer users *still* don't understand the basics of computing. I've had people who claim to be 'computer literate' refuse to send me a document in plain-text because they can't figure out how to do 'advanced' stuff like that. They just use all the default options of their operating system. They've got no concept of the issues behind using one document format, or another, nor of the issues of downloading stuff randomly from the net and installing it on your computer.

2) Most computer users will still run a mysterious program they get sent in the mail, or click on a link that promises to show them something 'fun'.

3) We've created a monoculture of operating systems. One operating system rules the planet, allowing hackers to focus all their effort on that one system. This is equivalent to everyone on the planet using the same locks, even with each lock taking the same key.

4) The operating system we've selected for dominance is, and always has been, radically insecure (though it has started getting better recently). Indeed, most computer software is radically insecure, using unencrypted network connections and storing personal data unencrypted on disk. Why? Because:

Security doesn't matter.

No one cares about security. Users don't care about security. If there is a new, cool widget out there with some security issues, users will download it and install it. When you try to point out to them that it might be making them more vulnerable to attack, they will look at you, blink, and carry on with what they are doing.
Because users don't care about security, so software writers don't care about security. Security features don't sell. Pretty graphics and animated icons and 'cool' is what sells. Until users start to care about security, *really* care about security, security won't sell.

Consider Active X. There is no good reason that I can think of for web-masters to still be using active X controls, but they still do. Active X is a means of sending little programs to run on your computer, which is an inherently insecure thing. Java does the same thing, but tries to make it more secure by not allowing the program to write to disk or gain access to other system resources. Flash is another approach. Most of the time these technologies really aren't useful, they are just there to do flashy animations that the web-pages don't really need. Every sensible user should block these by default.
But many people let them through by default, because constantly being told "Please enable flash...", "You need to download an active X.." etc etc, gets annoying. And, sooner or later, they will come across some 'cool' thing that they've just got to have access to. Maybe it will be a youtube video, maybe it will be Angelina Jolie naked, maybe it will be a little online game. People don't really *need* any of these things, but they are generally prepared to compromise their system security to get at them.

Consider firewalls: People pull down their firewalls. Back when I used to frequent IRC, I frequently came across people who weren't running a firewall, because it "Got in the way." Sometimes I'd connect to their machine and tell them what I could see on their hard-drive, in order to put the fear of god into them, but most times I wouldn't bother, they weren't going to listen anyway.

Consider passwords: People accept the 'Do you want me to remember your password for you?' option on web-browsers. This means that somewhere on their computer is stored the password they use for a website. Is it stored in encrypted form? No, of course it isn't! You are forgetting the rule "Security doesn't matter". If it were properly encrypted, you'd have to use a 'master password' to access it. Any encryption system that doesn't ask for a password, isn't real encryption. But people don't want to have to remember passwords, any password, so they'll always go for storing their password insecurely on their laptop.
I've known people to store their passwords this way on publicly accessible computers, so they can always come back to the same computer and log right in.

Consider email: People send each other games and 'dancing baby videos' (are you sure it's a video, and not a program?) in email. No one ever stops to think "Hang on, can I trust this game/video/whatever?" No, they just hit send. I've seen companies in which the staff email this kind of junk to each other across the company network. Then, some weeks later, the company gets a computer virus. Everyone is SOOOOOO surprised.

I could go on, and on, and on, and on. The reason cybercrime is so prevalent, and so *easy* is the users. And this isn't just ma-and-pop who don't understand computers, this is corporate interests too. If security gets in the way of anything that someone can claim, no matter how ridiculously, is interfering with the work of the company, security loses, and is taken down. If security requires users to remember passwords, it loses, and is removed. If security gets in the way of people seeing funny pictures, or dancing babies or having some cool new feature on their desktop, security loses, and is taken down.

Security always loses.
Hackers always win.

Why Open Source needs another decent Webbrowser

2009-04-18T16:31:00.000-07:00

The Open Source world needs another decent web-browser.

Why?

Well this ties up with a lot of issues related to how open source, and particularly linux, has been changing over the years, and how, in many ways, I believe that open source is becoming increasingly irrelevant and less open.

Way back when I started using linux the arguments in its favour were hard for the windows using crowd to understand, but there were some that they could get their head around.

1) Linux was fast. I'd start up applications, and they'd pop into existence on the desktop. They might be primitive looking compared to what you'd get under windows, but they did the job well enough, and people's eyebrows would raise when my PC booted in ten seconds flat once it had got through the bios stage.

2) Linux was small and undemanding. You could fit it on almost any piece of hardware more powerful than a 486. Old kit that windows users were unable to use any more worked fine for linux, and to this day I use a bunch of circa 1995 AST laptops running linux, so that I can take them anywhere and if they get stolen, what have I really lost?

3) Linux was reliable. It didn't lock up or crash like windows. There was no 'Blue screen of death', or if there was, it normally meant you had hardware problems, like your processor fan had stopped turning, and the CPU had thus overheated.

4) Linux didn't get viruses/trojans/hacked. This had a lot to do with it being simple and primitive, meaning fewer openings for exploits, and also that it didn't try to do everything for the user (and hence be too ready to open files and run programs that really it shouldn't be messing with). If you did 'ps ax', you could look through a short list of processes, and know what each of them did, and spot any that shouldn't be there (and I once detected some naughtiness going on with just this method).

There were other features that they didn't understand, but which were very important to me. I could get the code for programs, and compile it myself. If they didn't do some little thing that I wanted, or if they blew up unexpectedly, I could go in there and try to fix them, or add the thing that was lacking, or sometimes even take features out that, for one reason or another, I particularly didn't want. I could, and did, learn lots from looking at other people's code, and how they did things.
What's more, there were generally many programs available for linux to do a given task, each with its own advantages and failings. I could pick the one that was most right for me. When once, I heard another linux user saying "You should just use program X, and no other", I looked at him aghast and said "In that case, I should just use windows and be done with it. Linux is about choice. If it's not about choice, it's not about anything."

This was what open source meant for me. Oh yes, and it was free too, which was the main thing that the windows users REALLY understood. But for me the ability to choose, modify, understand and take ownership of the programs I ran, that was what was really important.

And it's all gone to hell since then.

Basically gtk and qt and Gnome and KDE came along, and linux slowly became windows. GTK and Qt are 'widget sets', which means they are libraries of buttons, and edit boxes, and progress bars, and what have you. Unix has always offered a choice of these to developers. But gtk and qt became the two 'big boys', and big they certainly were.
Over the years I've had cause many times to compile gtk, and every time it's gotten harder to do. Every time the results have been bigger, fatter and more bloaty, and as any self-respecting geek will tell you, the more bloaty software is, the more untrustworthy it is. Gtk now has a distressing number of sub-libraries that you have to compile to get it to compile itself, and I actually find that compiling that other monster of the UNIX world, the X-windows display system, is quicker and easier than gtk. In terms of memory demands these widget sets are monsters, so item 2 of the list of linux advantages is already gone.
The resulting programs are slow too. When I start up opera or firefox, there's always that period of time when one gets to thinking "Is it really starting up, or has something gone wrong? Oh no, I can hear the hard-drive churning away, it's still starting up." So, advantage 1 is gone.

GNOME and KDE are the 'desktop environments' that are built upon either GTK (for GNOME) or Qt (for KDE). These are collections of programs that all work together to produce the look and feel of the operating systems desktop. At work I've got a KDE desktop. I can't speak for GNOME, as I've never used it, but I can say that once I started using KDE, I started seeing crashes and lockups. Windows users would look at me, grinning and say "I thought linux didn't do that?" and all I could say was "it didn't... used to."

And they'd ask me "Why are you using it?"
And I had no answer.

Oh, it's still got the edge in many things. Microsoft are always able to stay one step ahead by producing something like Vista that's ridiculously demanding in terms of memory and processing power (Two GIG? Are you sure you mean gig, and not meg? TWO GIG?) slow, broken (Webdav, anyone?) and unhelpful (What do you mean my old programs won't run anymore?). But linux is seriously catching up. Nowadays, linux demands way too much memory and processing power (the stock answer being "well memory is cheap"). I find myself having to reboot, which once I never did. Programs are slow to start, and sometimes slow to use ("Well, you need a more powerful computer"), and it's hard to find programs that do useful things, and don't require the two great monsters, gtk or qt ("Well, you should use the STANDARD programs, not something else"). What's more, gtk and qt are such monsters, that it's meaningless to look at their thousands and thousands and thousands of lines of code, and I generally wonder how many people really understand it. When I do a "ps ax" under KDE, I see legions of mysterious programs running that are presumably part of KDE. I have to use an 'official' distribution (like RedHat, Suse, or Ubuntu) of linux if I want things like firefox, because I can no longer get them to compile on my own, so I can't make my own distribution of programs that I've selected.

Linux isn't small any more.
It's not fast.
It's not reliable.
It does crash sometimes.
I can't make sense of a lot of it.
I can't make my own linux from basic parts, as in former times.
I think it's probably not as secure, it's so big and complex now that there must be a thousand nooks, crannies and holes for a hacker or virus to hide.

So, what is the advantage, exactly?

Well, I can still make my own linux from basic parts, and I do. And it's generally fast, and it generally runs on low-end hardware, and I've got far better control and if something breaks, I can generally fix it. But there's one vital thing that it lacks.

A decent webbrowser.

Because if you want a webbrowser, then you need either gtk for firefox, or qt for opera/konqueror, and then you're back to a windows-style linux. Why not just use windows, and I.E. and be done with it?

So, when google chrome turned up in the windows world recently, I was at first infused with hope. Who knows what widget set they would use? But, google chrome for linux still hasn't turned up, and if it ever does I've come to realise that they'll almost certainly use gtk or qt, not something small and fast like fltk or even, perish the thought, one of the old motif clones. So, it will be big, fat, slow, and demanding. It'll be no use for me.

The webbrowser I use most on my home-grown systems, is links. Links stopped really progressing as a browser some time ago, but it's about the only lightweight graphical browser that can handle some javascript, can kinda handle frames and iframes, etc etc etc. It's admittedly got a very clunky user interface. One is often struck how, with just a little more work, it really could have been a much better user experience. But it starts up fast, and it's fast to use, and it will run on old systems with no memory, and it works even in text mode as well as graphical, allowing me to use it over ssh links, and maybe allowing blind people to use it with braille keyboards.
However, a while ago the links team said that, in their next release, they were going to take javascript support OUT. So, it's actually stepping backwards in terms of its important features. More and more websites require javascript (often for mysterious reasons. Click on this link, and it triggers some javascript that... takes you to another web-page. That's odd, why couldn't you have just used and old-fashioned straight-forward html link to do the same thing? What does the javascript do that I don't know about), and other than firefox, opera, and konqueror, all gtk/qt based, there is no browser that offers these.
Don't be telling me about the 'alternative' browsers like Galeon or Skipstone. Most of these use the core engine from Mozilla/Firefox, so they are just Firefox in drag. They've got the same demands and requirements, so I can't easily build them on my systems.

What I want from open source these days, is a small fast web-browser that does most things, not everything, but most. That somehow handles frames and iframes and javascript, and yet doesn't demand me to buy a new computer just to run it.

Google Chrome for linux could be that browser.

But it won't be, will it?

Blogger API: I'm impressed

2009-02-22T03:48:00.000-08:00

Well, I'm impressed.

I've managed to figure out the blogger A.P.I. sufficiently to write a little command-line app that lets me type up my posts in an editor (vi) and then post them to blogger. Why would I want this? Well, my laptops are weedy things that I got for free, and which date from the 1990's. Full-featured modern browsers run painfully slowly on them. I normally use 'links' as my preferred web-browser. Now, you look at links, and yes it's clunky, and yes, the pages don't render as they 'should', but it *mostly* works. It's small and fast. Then you look at firefox, opera, konqueror and co. They work on many more sites, and have more features, but to get this extra 10% of functionality, they are utterly huge and slow with masses of dependancies. Something isn't quite right with this picture, I feel. But this is a rant for another day.

For me, writing my posts in 'vi', and then sending them later, is the ideal solution. It's quick and easy, and 'vi' recovers it's data effectively after a power-cut or what-have-you. However, most things these days, are hideously overengineered, and if a company decides to let others 'hook into' their systems, it usually involves downloading and using libraries, and before long your system is littered with libraries that you use for one app.

So, finding documentation that tells you 'connect to this address, send this HTTP transaction, you have now posted to your blog', is damn impressive. This is the way it should be, open, and fairly simple. Took me about an hour to get it working. All hail google.

Colum

Cory doctorow on the history of copyright

2009-02-19T08:37:00.000-08:00

Here (you tube) is a must-see public speech by Cory Doctorow explaining the work of the Electronic Frontier Foundation, and the politics surrounding DRM and copyright. Pity the sound quality is poor.