Monthly Archives: April 2014

Cobbler

I finally clear out enough bugs and issues that I could write the first version of the Shoe Cobbler. It’s just a screen (app) built into Shoes that allows you to manage your image cache. Not that big a deal unless your image cache is corrupted.

In Shoes 3.2 (federales) there are two kinds of Shoes. Loose Shoes and Tight Shoes. You can only have a loose shoes if you build from source. If you want to poke around inside Shoes, it’s the way to go. Tight Shoes is like the older Shoes 3.0 and 3.1 binary downloads — only Shoes 3.2 might actually work for many Linux variants and the Windows downloads runs. Tight Shoes has it own copy of Ruby and it’s own gem directory. It can’t be use your existing Ruby Gems (if you have them). From 3.2b1, Tight Shoes 3.2 had a method to Jail Break and use gems that Shoes didn’t install. A clunky ‘edit this file with notepad method’ that I’ve not mentioned much because it was ugly.

Cobbler changes that. for Tight Shoes, Cobbler will allow you to enter your existing gem directories. Or edit the ones you set in Cobbler and Tight Shoes will look in those directories for gems. Jailbreak. Of course if your existing gems are compiled for Ruby 1.9.3 (say) and Shoes is Ruby 2.0.0 that might be your problem to sort out. samples/simple-info.rb is your best friend.

There might be Shoes 3.2b6 downloads if you look. Not an official annoucement.

Simples.

This post and some future posts may not seem related to Shoes but it will be become clear.
Besides I’m tired of chasing internals shoes. Time to step a way for a few days.

Now that I have a modern real Intel (Tm) box, I can install OSX. I’ve just never bothered.
So I installed 10.8 Mountain Lion in a VirtualBox. It works. It could work better (screen size) if I spent some time with VirtualBox. You can find plenty of youtube videos for installing OSX in a VirtualBox. Hackintosh.

I didn’t know about VirtualBox’s newer networking capabilities that allows the host os and the guest os to network so in theory I could nfs export my shoes directory on ubuntu and nfs mount that in OSX. That would make Shoes development for OSX easier. Or rsync between them (not as good as nfs).

Since that’s a lot of work I don’t want to repeat it for 10.9 Maverick. So I’ll install 10.9 and the proper xcode (I hope). And then rvm and ruby and homebrew and then just possibly, maybe, also build GTK3. It sounds like an exercise in frustration and it might be. It could be more fun than updating Ubuntu and the Gnome Shell/Cinnamon mess.

—- later, many, many hours later —

It’s a wonder that I got 10.8 installed considering the crap that 10.9 involves. The VirtualBox networking is a deep dive since DNS seems not work in some situations. Fortunately, I can play with that after getting 10.9 installed. The 10.9 I have is not an .iso, it’s a bootable(!) .dmg. Linux doesn’t like that. Tools to convert to iso don’t work for me. Why not move those file to 10.8. Surely it knows what to do with them?

What I need is way to move files from Linux to the OSX VM. I can ssh/sftp to my hacked Buffalo NAS. It’s slow when moving 2GB and 5GB files. It works but my poor little NAS.
Meanwhile, I attempted to get a 80GB USB drive setup. That was bit tricky. Things got a lot better when I added myself to the vboxusers group. Eventually I formatted it in Linux as hfs+ and then as root, from the command line, cp the files, unmount from Linux. startup OSX 10.8 and it can read them (slowly). Can I install xcode 5.0.2 on 10.8? It tries. And Fails. But 10.8 is a detour. Right?

I want to get that 10.9 img onto the USB drive so it can use it to install in a new VirtualBox VM. Disk Utility – just a unintuitive to me as it was back in 10.2.8. Delete what’s on the USB drive. Image the 10.9 dmg to it. Simples? Did I mention Gawd Awful slow USB? Do I know what I’m doing? I don’t think so.

[Update Apr 23, 2014]
The experiment is over. Too many road blocks and dead ends. If you want an Apple machine or experience, get a real one. I’d forgotten what the Apple ecosystem was like. Now I remember. I’ll remain a happy APPL shareholder but not an Apple developer. I have enough challenges.

Replace curl – part 4 – The end is nigh. Shoes 3.2b5 is here.

Really, really near. The only hard part left is fixing the .h and .c files and the rake files that curl is not needed. Just a small matter of #ifdef that could take hours or days to get right. It’s working for the Raspberry pi build. It’s working for Windows(Gtk2) and it’s working for the linux source build (to dist/) and those are the ones I care about the most. Also working for X86_64 and i686 generic Linux. I have no interest in the Windows native widgets (msw) variant, when the Gtk2 variant is working so well on Windows. For Red Shoes, gtk is the future.

Shoes 3.2b5 is available if you click on downloads menu and pick the one you want. “What does b5 bring to the table” you might ask? I got rid of curl and sqlite. But you don’t care about that unless you’ve tried to compile shoes. I got image download caching working again. It’s just better. Just slightly better, but it was hard work for me!

I’m even going to clean out the eariler beta downloads by moving them to old.

What’s next, in b6? I don’t know. I’ve got a butt load of clean up I could do, have to do, some day. The binject and packaging could wait forever. I don’t know. Maybe I’ll move on to Shoes 3.3 (gtk3).

If you choose not to clone from my github, there is a source tar.gz in the downloads.

Replacing curl part 3 – da caches

One of the Shoes 4 folks thinks my changes to download.rb doesn’t stink. Good for us. I’m not competing with Shoes 4. I’ll contribute when the internal venn diagram say I’m overlapping. Tis only fair – I used their download code to start with, I added to it, I should let them know.

The shoes command ‘image “http://a.website.com/funny.jpg”‘ will download the image and place it in the current slot (aka canvas). In Red Shoes, there are thousands of lines of code if you count the windows C interface, and the Linux curl interface and OSX objc interface. I don’t want that for Shoes 3.2 if I can get rid of it. My love of embedded curl borders on rage. It may not be rational but there it is.

It turns out that a lot of those lines of C have nothing to do with downloading and a whole lot to do with cacheing the downloaded images so they don’t reloaded from the file systems or from the network if used again. As best I can tell, Shoes 4 doesn’t have a cache for the downloaded images and I’ve thought hard about whether 3.2 should. I thought even harder when I saw that the download cache depends on Sqlite. I don’t distribute sqlite with Shoes 3.2 and I’m not going to unless I really, really have to.

Shoes uses Sqlite to provide a persistant cache that lasts between or across different Shoes invocations and even different programs. It’s a worthy thing. I think I can replace the Sqlite in data.rb using the built in sdbm gem. For now, data.rb says “don’t have” when Shoes queries that cache. It would be hard to debug the download if it’s already in the cache and my download never gets called.

So I added a rbload.c to shoes/http/ and modified the default line rakefile env.rb to compile it instead of shoes/http/curl.c There’s only 4 functions to implement in C by calling into the same download.rb I used for the download command. If done properly I can get all the Shoes per-process caching and the inter-processing, persistant caching.

Of course there are many ways to get this wrong. I’ll find some of them.

Replacing curl part 2. Shoes 3.2b5

Yes, a new beta is available for Windows (gtk2) and raspberry pi. It fixes problems with the Shoes download command on the pi and possibly windows. I also added a simple-download1.rb to samples to test some things that simple-download.rb doesn’t. There are
more tests needed and possibly some coding as well to handle incoming options and dealing with http headers but frankly, if you really need that ability just write your own open_uri code. That’s all I did.

Getting rid of curl can now proceed with the more difficult task: Changing the C code in image.c and the dance of hell with the .h files and rakefiles. Testing should be easier.

One final note for Windows. It’s entirely possible that Shoes will hang on internet operations because the Firewall is letting the Shoes you installed make a connection.

Replacing Curl – part 1 of many

Let the whine begin. Aka, bug report with attitude. Shoes-3.2b4 for Windows craps out running sample/simpler-downloader.rb. On the pi, it does a segfault. shoes/http/curl.c and shoes/http/download.c,winhttp.c. Works fine for x86_64 and i686. The last thing I want to do is debug either of the failures. I could (have done before) include curl in the Windows builds. Every time I look at curl configuration and curl internals and the shoes code for use it, I shudder violently. If there is a better way to do downloads now than when _why wrote all that code, I want to find that way. Shoes downloads in two situtations – a uri in a image load and a call to download. I’ll fix the download method first and then the image and then if that works, do a major overhaul of the .c and .h and all the rake files to remove curl/winhttp. If I can.

I’m borrowing from the Shoes4 code – their api is the ‘future’ although they claim it is based on 3.1. I’m going with what it is in the Shoes Manual unless it’s too obscure or annoying. If there is API breakage so be it. I’ll create a download.rb in lib/shoes/ and require it from cache.rb (not the best place – it should be part of Shoes.rb – but who cares?)

So, lib/shoes/download.rb looks like this
# My ruby downloader move code to shoes.rb or lib/shoes/ when working
class Shoes
class Download
# need headers, body, status @vars as well as the Shoes4 stuff
attr_reader :progress, :content_length, :headers, :body, :status
def initialize url, opts, &blk
@blk = blk
start_download url, opts
end

def start_download url, opts
puts "Download from #{url} with #{opts}"
end
end
end

# Monkey patch over the 'C' code.
class Shoes::Types::App
# Shoes::Types::App seems wrong but it works.
def download (url, options, &blk)
Shoes::Download.new(url, options, &blk)
end
end

When I run samples/simple-downloader.rb I get

Download from http://shoes.mvmanila.com with {:save=>"shoes.mvmanila.com", :progress=>(Proc |a|), :finish=>(Proc |a|)}

So I can replace ‘C’ code with Ruby code. I knew that should work but a test is needed to prove it. Will the rest of the Shoes4 code work? I suspect not but it’s a good place to start from.

Maybe not the end of Shoes 3.2

I love it. A helping of irony is always tasty. I announced that I’m going to stop working on 3.2 and move onto Shoes 3.3 using Gtk3 and no one should expect another Shoes 3.2b4 or higher and nothing more if Shoes/Windows/Ruby won’t play nice. So I updated the rake files — its been a while since I built and tested the MinGW/GTK variants (back when I thought Ruby 1.9.3 was the baseline for Shoes 3.2). Guess what?

They work surprisely well !?! The Gtk3 variant has the same Gtk3 bugs as Linux – no surprise there. That’s what Shoes 3.3 is supposed to fix. So many of the samples works as expected in Gtk2 without hanging Shoes that I started poking around with the gem handling (samples/simple-rubygems.rb). It downloads with progress bars that move. Yay. It installs pure Ruby Gems. It fails to build bluecloth – as expected since my Win7 doesn’t have the compiler needed. Just like Linux. So I tried nokogiri which we learned has a mingw binary payload available. Downloaded! Installed! Dies on the ‘require’. That could be a Ruby version mismatch since I’m using 2.1.0 and it could be using 2.0.0. So why does simple-downloader.rb fail? Because it uses an very old bit of complicated C to work around Ruby http problems and newer Rubies have fixed that problem? Could be, grasshopper. That’s a fixable and perhaps I could even get rid of Curl requirements on Linux. That would be a win-win-win.

I also found the hang on Windows. It’s not Ruby and it’s been reported before and forgotten. If you put a window in front of a shoes window (or partially cover it enough), i.e. Shoes goes into the background stacking order then it will max the cpu until some Windows watchdog thinks it needs to be stopped. YES! It’s not a Ruby threading problem or a Gtk threading problem. Finding and fixing it may not be so easy but it won’t require changes to Ruby or Gtk. I might be able to fix that. That’s a huge win for me. I’ve uploaded the MinGW-Gtk2 3.2b4 version if you want to play with it.

Both bugs, the windows expose event handling and the url downloader are show stoppers for Shoes/Windows. I suspect the first is a simple fix but hard to find for the Windows ignorant and the second requires ripping curl/windownload out Shoes. Hard to say which one I’ll tackle first. Could there be a 3.2b5? Of course there could be.

Nearing the end of Shoes 3.2

I’ve been thinking about what to do next. Getting packaging working is a really deep hole and frankly not worth that much effort unless Windows and OSX have working Red Shoes packaging too. That means GTK3 for all platforms and that means dealing with the Shoes/Ruby/Windows threading problem now (and fixing the known gtk3 bugs). If the thread bugs are too much for me, then the next Shoes 3.3 is not possible.

Yes, there are some minor bugs in 3.2 for Linux but they are really minor and have work arounds. In the previous post I showed how to use Shoes to install the serialport gem on a Linux system without any Ruby but the one inside Shoes. And then I showed how in install the Sqlite3 gem. Shoes binary downloads (the .run) work much better in 3.2. Much better than they ever did.

So that’s my plan. Get Shoes-3.2/GTK3/Ruby-2.1.0 working on Windows w/o the threading bugs. If I can find a solution, then there is no reason to care about packaging and other enchancements.