Monthly Archives: June 2014

What to do with Windows

I found a blog post about using Gem’s Gzip::TarReader. Sweet. And easy enough to adapt for my purposes. It’s a huge speed improvement over Minitar::Shy.xzf

The packaging tests was going well until I got to Shoes/Windows which just hung when trying to pack a Linux .run or and OSX .tgz distribution. Windows doesn’t have a symlink available to Ruby. Why it hangs instead of throwing an exception is a secondary problem (probably a method_missing somewhere in Shoes). It turns out OSX didn’t need that symlink, so I no longer create it. OSX packaged but it doesn’t run because the +x bits aren’t set on some of the shell scripts and shoes-bin and others. Same problem for packaging for Linux from Windows. It should not have surprized me that Windows doesn’t maintain permissions. Everybody knows that, right?

That’s a game stopping oversight. Time for plan A, B, and C.

(Plan A): It would be relatively easy (as these things go, [snort]) to go back to using Binject for creating .dmgs for OSX which creates in memory hfs+ file systems.

(Plan B): try writing some lines of shell script that set the permissions of the known files and cause the launch script (which may not be +x) to execute them. That would work for Linux because of how it’s launched but is’s hacky and patchy. It also won’t work for OSX

(Plan C): Just ignore the problem. I don’t really care about Windows (they can package for Windows just fine) and as I’ve said before, packaging really shouldn’t be used, although I’ve weakened my argument by making the download and extract run fast enough to not suck.

Plan A also could be used for packaging Linux. Create a dmg, uncompress and put the linux tar ball in it (in theory it will save the permissions bits and I could fix it if it doesn’t – it’s only C). Then [wave hands] dump the +hfs with a GZip::TarWriter [wave hands again]

Or I could use plan B for packaging for Linux and Plan A for packaging for OSX or maybe something else.

Did I mention Slow with a side of bugs?

I start on that 5 x 5 testing of packaging. Linux can package all 5. OSX can Package all 5. (I don’t have to do tests of every Linux since for Shoes, they’re all alike.) Here’s the screen in slow motion.

packaging

Windows has several head scratching issues. It can package for Window (they all do that) but it doesn’t want to untar any of the Linux or OSX distributions. I saw that hang before on Linux when I was re-implementing and then I did something and it went away. It’s back. There’s an even more puzzling problem that might cause the first one. Isp.exe (any source) will install a Shoes in the system (and menus). That’s not what I want but the huh? problem is that the Shoes is 3.2.10, not the 3.2.11 in the payload. Which means that Windows is hiding on old version of Shoes where I can’t find it – yet.

Shoes 3.2.11 is available

Shoes 3.2.11 is available for download. Changes from 3.2.10 involve limited packaging support.

Shoes 3.2 (Linux) can package a script for OSX (10.9+), Windows 7+, and Linux (i686, x86_64, armhf). Before anyone does a ‘happy happy’ dance let me point out several issues that I probably won’t fix anytime soon, if ever. This may not be the packaging you remember. Allow me to use an example. I have a Shoes script, isp.rb. Using a Shoes 3.2 that I’ve download (or built from source) for my platform I can create an isp.exe (Windows), isp-osx.tgz (OSX) and Linux versons isp-x86_64.run, isp_i686.run and isp-armhf.run. Each one of those is 15MB to 20MB in size because it includes Shoes. None of them will install Shoes (maybe the Windows could but it shouldn’t). Command line jockeys might be able get normal Shoes working from the isp.xxx but why bother? Download the real Shoes if that’s what you want. OK? Please?!

You will note that download is likely to be faster than packaging. Packaging is seriously slow when dealing with OSX and Linux. I do mean seriously slow. Really slow. You could have packaged up a .shy with your script, upload that and write a blog entry before you get isp-osx or isp-linux built. Many minutes for each variant and then you’d have to upload them for users to download.

Perhaps you’re thinking “if it’s so crippled then why did you get it half assed working, Cecil? Huh? Why?”

Because I could and nobody else would.

There’s also some end use cases where this is exactly what is desired. For example, I’ve got a Yield To Maturity shoes app I wrote in green shoes. It’s a very tricky calculation that you’re supposed to call your stock/bond broker to do for you. They probably use some Excel macros liked to their inside databases. Perhaps they’d like a cross platform native gui app? OR not, but some of your edge cases might.

Don’t be alarmed if the Shoe 3.2.11 dates and sizes change in the next week – it’s only me fixing packaging problems. I’ll let you know here is something important has changed with Shoes 3.2.

After the fun wears off

I’m not complaining, I’m just communicating. The good news is that I can package an Windows exe from Linux (and all three Linux architectures). Easy peasy as I expected. I also discovered that OSX is quite happy unpacking .tgz downloads as well as the previous .tbz. So that’s what I’m going to produce. That’s important info for a reason.

No one likes BINJECT::dmg. It’s a cess pool. As i mentioned in the previous paragraph, tgz works just fine now days. The OSX handling could look a lot more like Linux (with plists and so on). The bad news is that packaging FOR linux uses the pure Ruby Minitar code inside Shoes and it is butt ugly slow when it’s 15MB to unpack. Probably because no one ever packed a Shoes script for linux with linux since It was well known not to work. That works now and it’s slow.

Why would I inflict that overhead on OSX packaging? Because packaging has little value other than the cool factor. Are you really going to pass around 20MB .exe/.app/,run when the user could download and install Shoes permanently and then download a small shy of your script? Another cool and impractible idea is to set the minimum wage to ‘livable/sustainable/fair’ and pay for it with magical green unicorn farts, Packaging is not sustainable. I may get it to work again because I’m pig headed. Butt ugly and slow is good enough for something you shouldn’t be using.

I might also discover that gem:: can read tar balls or maybe new zip specs are good enough. We’ll see.

Why packaging so damn cool

On my Ubuntu box (64 bit) I packaged the isp.rb script for Linux-i686 (32 bit) and copied it to my Debian VM. In Debian I did ccoupe@elvis:~$ ./isp.run --target foo which puts Shoe and isp.rb in the ~/foo directory and runs isp.rb. That’s nice.

Then on the debian system, I did ./foo/shoes -p That runs the packager (I could have run it from the splash screen, without the -p). Then I packaged the ~/foo/isp.rb script and created a Raspberry pi version (isp.run) which I copied to the pi and and executed it with ccoupe@pi ~ $ ./isp.run --target foo. Even when Shoes 3.0 and 3.1 was working you couldn’t move between different Linux architectures.

There is a bug in using the pi to create an x86_64. It may be the pi’s odd Locale settings but that’s why I do these gruesome tests. When it’s fully working you could create Shoes/OSX version of your script and use that to create a Windows version or … Actually it’s a 5 X 5 matrix which is a whole lot testing for the anal compulsive. Thank god I’m only a hacker and I don’t care about perfection.

You will see shoes 3.2.11 versions in the download directory. You shouldn’t use them for packaging until you read about it here. OK?

My rules: Linux first

I’ve got the packager working just barely enough that I tried packaging a script for the Raspberry. It’s a little program I wrote a few years ago to ping the dns servers of my ISP and log the succeed/fail times. It’s called isp.rb It’s not overly clever and it only has value to me but it’s a good Shoes script for testing packaging. Older Shoes packaged it up as isp.run. It’s a multi-arch world now days so I need to change it to output isp-x86_64.run, isp-i686.run and isp-armhf.run. I haven’t done that yet. (if you have an ARM Windows machine please contact me).

What might not be well known is that .run files actuall have help and info and other command line switches so ./isp.run --help or --info or --keep can help debug and packaged script install problem. Of course I found one. Somewhere in my notes or maybe it was the wiki I noted ‘this will break’ on future rubies. And it did. It broke when Shoes went from 1.8 to 1.9 and was ‘fixed’ for 1.9. The real problem is that Shoes has a copy of minitar and minitar doesn’t handle symbolic links so libruby.so.2.1 doesn’t point to libruby.so.2.1.0. The shoes launch script fixed that for 1.9.x (for packaged shoes). It a ‘oddness’ of Shoes packaging. Clearly I can and will fix it for 2.1.x with a hard coded hack to replace the 1.9.x hard coded hack (I wrote the first hack so I’m allowed to hack it again). Grandfather Hack rights if you will. It’ll break again when Ruby and Shoes moves to Ruby 2.2.x

Packaging has those oddities. Different ones for Windows, OSX and Linux. It one reason developers are afraid of it. In Shoes 3.2, Packaging allows multiple architures for (.run) and easily extended for .exe and OSX. I plan to only support 5 arches (with all the OSX variants it could easily be 6 more – no thank you). What Shoes 3.2 won’t do is packagee ‘download if needed’ which is what most people want to do. Shoes 3.2 will disappoint them.

I suspect Windows would be a lot easier to get working for Packaging [as I’ve defined it] but I care about the Linux and the Raspberry more so that’s first.

Begin the packager

I do not know how this is going to end. I don’t know if I should have even started it. There may be brick walls I don’t to surmount. Then again, self packaging is a really cool features of Shoes and I think it too damn clever to not try to make it work again.

early-packager

That’s the new packager screen. Compared to the old screen, it’s really minimal. No options for ‘download if needed’ or video/no-video. No options at all. No checkboxes or radio buttons. Push the button you want. It downloads, unpacks, inserts your app, repacks and you get a something you can upload for others to download. Push another button and wait if you want something additional. That’s the good intentions.

It is not finished, or even working yet. With the changes I’ve made to Shoes 3.2, it may not ever work. It certainly won’t work the way some folks remember or want. Some plaforms download the whole mess and some will download it when the ultimate end user runs their app.

There’s some new infrastucture around those buttons. First, they are created from the execution of a cgi script (that just lists the Shoes available for download) Look at the OSX button. Hey, it’s 3.2.11 and the others are 3.2.10! There is an osx 3.2.10 file available but you only get the latest. Magic Cloud computing! Depending on the details with Shoes 4, it may be possible to offer that as an option. From what I’ve read but don’t really remember it might be very easy. Of course that wouldn’t work if you’re Shoes app depends on some binary gem that Shoes 4 can’t handle. Why let a few pesky facts get in the way of a dream?

Until the facts puncture the dream balloon, I’m going to try.

[Update]
I forget to mention, it’s not deeply tied to my website. I added an option to Cobbler so any one can enter different urls for the cgi script and download directories. It just defaults to my website but any website that can run the following script would be usable.

#!/usr/bin/ruby
lns = []
Dir.glob('../shoes/*') do |fp|
tm = File.mtime(fp).to_i
sz = File.size(fp).to_i/1048576
fn = File.basename(fp)
lns << "#{sz} #{tm} #{fn}n" end require 'cgi' cgi = CGI.new cgi.out{ lns }

Unless I change things.Which is likely

Shoes 3.2.10 – not a beta

I invite you to download Shoes 3.2.10 and give it try. Please let me know if it works for you, or if it doesn’t. The download tab above will give the the simple instructions. You can easily delete it if you don’t like it and it doe not interfere with any other version of Shoes you might have. Shoes 3.2 is available for Linux 32 and 64 (intel/amd), Raspberry pi (raspbian), Windows 7+, and OSX 10.9.

What’s new from Shoes 3.1?

  • Linux binaries that work. OSX is working. Windows works too!
  • Uses Ruby 2.1.x and Gems 2.2.2
  • A new Cobbler screen allows you to install and delete gems for Shoes with a GUI. Developer level folks can tell Shoes to use any existing gem directories – if they are Ruby 2.1.x compatible. You can also make a copy of the Samples so you don’t have to poke around inside dot directories or navigate to weird places to find them.
  • There are no built in gems. No bloopsaphone, or json or hpricot or sqlite3. You can use the Cobbler facility to down them or your favored equivalant.
  • Linux users get Menu item in their GUI. The Shoes icon has been changed so you won’t confused it with other Shoes.
  • Sample code that uses hpricot or sqlite won’t work until you fix them or download the gems.

Under the covers, in the source code and the rakefiles, a lot has changed to make it easier to maintain (my opinion). MRI Ruby 2.0 and 2.1 drives the bus and Gem’s are the backup driver. Shoes 3.2 just rides along where they go. The 3.0 and 3.1 code tried to convince Ruby and Gems that they can bend to Shoes’s will. I also do a lot of things to enable cross compile magic. It only takes 5 minutes of wall clock time to build all 5 versions of Shoes. It take four times longer to upload them and many times longer to download them into each Virtual Machine or real system.

There’s also a default option for building from source if you don’t care about building Shoes binaries to distribute. That version can’t be copied over the internet since it’s linked to your ruby and your gems installation (and often your user name or Linux variant). It’s smaller in size, builds faster and it’s great for code jockeys. Shoes knows not to let you package that. For most folks that’s the right thing.

We don’t need no stinking decprecations

You might have thought it was impossible but I’m actually learning Objective C and Cocoa and AppKit and and and and. I decided to fix deprecations in cocoa.m first. One of these days, deprecated stuff from 10.5 will stop working. Font handling appears to be OK now. And not a nightmare of FSrefs and FSgetCatInfo and other Carbon bad memories I remember from Mac OS 6. (Yes, I do remember the time before Switcher) The new code is also smaller except I #if’d it so the source file is larger. Not a problem for me.

I’m also cleaning up the open/save file/folder dialogs (deprecations). That was on the list of annoying things about Shoes OSX. I’m not sure I can fix Save Folder the way I’d like but it’s worth a try and I’m learning. (it’s also funky in Linux and Windows too). I’m also starting to learn which Objective-C warnings shouldn’t be ignored. Even if Shoes doesn’t crash, some warnings suggest “that probably won’t work as intended”, and they don’t.

[update] The OSX save folder dialog no longer sucks. I’m pleased with my OSX progress. Who knows, I might fix a reported OSX bug in the days ahead. It’s possible. Or I could think about fixing packager now that I know almost enough to fix the launch/download stub.m. It would improve my OSX knowledge but thats future talk.

Shoes 3.2.10 will be the same as b9 with those minor fixes for OSX I just mentioned. I should announce it.

First Contact! OSX fix #1 of many

I don’t really know what I’m doing when it comes to OSX and Obj-C but I do know enough to learn more and not give up just because ‘it’s too hard’. I also know to learn with something simple, but useful and that would be shoe_font_list() which is called at Shoes init time and populates the Shoes::FONTS array for Ruby. The clang compiler complains about deprecations (from 10.5 or 10.6 time). Time to learn and update something that works.

tiny test script:

Shoes.app do
  para "Fonts:n"
  Shoes::FONTS.each {|font| para font+"n"}
end

I learned OSX folks don’t write as many blog posts describing problems and solutions like the Linux folk do. So, Shoes used the older ATS C level api and the newer better thing is CoreText (CT). Note, I wasted a day digging into a hole with NSFontCollection (but I learned some Obj-C so I can’t complain). I found this bit of code and I modfified it slightly for Shoes. After much testing and horsing around, it works. I don’t about the memory management issues. FIXME: I guess, maybe.

Since shoes_font_list is small, I’ll post it here for others who may google upon it.

VALUE
shoes_font_list()
{
  INIT;
#if MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_5
  VALUE ary = rb_ary_new();
  CFBooleanRef value= kCFBooleanTrue;
  int vtrue[1] = {1};
  CFDictionaryRef dict = CFDictionaryCreate(NULL, 
					    (const void **)kCTFontCollectionRemoveDuplicatesOption, 
					    (const void **)&vtrue, 1, NULL, NULL);
  CTFontCollectionRef fcref = CTFontCollectionCreateFromAvailableFonts(dict);
  CFArrayRef arrayref = CTFontCollectionCreateMatchingFontDescriptors(fcref);
  CFRelease(fcref);
  CFIndex count = CFArrayGetCount(arrayref);
  CFIndex i;
 for (i=0; i<count; i++) {
    CTFontDescriptorRef fdesc =(CTFontDescriptorRef)CFArrayGetValueAtIndex(arrayref, i);
    CTFontRef font = CTFontCreateWithFontDescriptor(fdesc, 0., NULL);
	CFStringRef cfname = CTFontCopyFullName(font);
	static char fname[100];
	CFStringGetCString(cfname, fname, sizeof(fname), kCFStringEncodingUTF8);
    rb_ary_push(ary, rb_str_new2(fname));
  }
#else
  ATSFontIterator fi = NULL;
  ATSFontRef fontRef = 0;
  NSMutableArray *outArray;
  VALUE ary = rb_ary_new(); 
  if (noErr == ATSFontIteratorCreate(kATSFontContextLocal, nil, nil,
         kATSOptionFlagsUnRestrictedScope, &fi))
  {
    while (noErr == ATSFontIteratorNext(fi, &fontRef))
    {
      NSString *fontName;
      ATSFontGetName(fontRef, kATSOptionFlagsDefault, &fontName);
      if (fontName != NULL)
        rb_ary_push(ary, rb_str_new2([fontName UTF8String]));
    }
  }

  ATSFontIteratorRelease(&fi);
#endif
  RELEASE;
  rb_funcall(ary, rb_intern("uniq!"), 0);
  rb_funcall(ary, rb_intern("sort!"), 0);
  return ary;
}

shoe_load_font(filename) will be next. No, I haven’t fixed a bug users would see. I have a lot to learn about Obj-c and Cocoa and AppKit. I’ll start with something bite sized before diving into the the deep end of the pool.