Monthly Archives: September 2014

Thinking outside the sandbox

One of the things I like about Shoes is you create a prototype gui pretty quickly and that helps me define the problem I’m trying to solve. Do I want radio buttons or checkboxes for this option? Doing that helps me think through the problem. It might help readers to see what I’m intending to do. As always, there are no promises, it’s a mockup, float the witch up the flagpole.

So, if the problem is Shoes packaging what would better look like? packaging-new

The first thing that’s note is the option to download Shoes or include Shoes. That gets us back to what Shoes Raisins could do and my current effort. I still have work to do on that for Linux and OSX but the Windows part is working and the the other stuff just need some “tweaking” to use a download url or two that works. He says. Here’s the Shy dialog to fill out the commentary below.
Shy-screen

Now we can talk about the future beyond parity with Raisins. If you click that Advanced options radio button then 4 options drop down. In case you didn’t know, when you package a shy (like my ytm example above), the shy container is always expanded – the user has to double click it again and it goes through a re-install if needed. Just to run your app. On low end Linux machines like the Raspberry pi, it’s particularly offensive as you suffer watching the marching ants across the terminal. Every time you want to run your app.

There’s another gotcha. anything your write in that “app” directory is going to be wiped out at the next launch. Yes, you should have written the Shoes app to store it’s values in ~/ or $HOME but you probably didn’t think about that. Ytm.rb was a Green Shoes app and has data files to read and write, right next to ytm.rb script. You might be thinking, “Cecil, you’re a lousy programmer if you didn’t think about that! You could have written code in Shoes that behaved better if it detected it was being reinstalled”. Actually, that’s two thoughts. On the first charge I plead guilty — I’m not that smart. (Hold on to that second thought).

If you package an app and select Advanced-> Expand shy in users directory’, a future feature that is not implemented yet, when the app is run by the user, after its does the Shoes existence check and installs Shoes if needed it uses Shoes to pop up a dialog to select a directory to store the shy in and all its data files. Yes, the user would have restart Shoes and pick an app (assume it yours) and if they wanted a menu entry or short cut to that ytm.rb looked like an app, they’d have to do that with what tools and skills they and there platform have.

“Wait a minute, Cecil. What script is run if you don’t run the users script on launch?”

Now you can bring bring out that second thought I asked you bottle up. It runs a simple Shoes installer script supplied by default when you chose ‘advanced packaging’ The default probably won’t or can’t do much more than asking for the directory to expand the .shy in,

Does that “own install script” button make sense, now? It’s a Shoes app. It’s Ruby. It can do any thing Shoes can do – like generate menus for Linux and run ‘sudo’ if you can figure out how. Or create Windows shortcuts somehow or unpack a tar ball of gems to install. Or replace icons. Or download something from you own website. I’d prefer it’s a not licensing key or a virus but I can’t tell you what to do with your custom installer script.

I’ll probably get around to doing the default installer script with ‘expand shy’ option because I want it. Anything beyound that is speculation. Gems can be tough. My thought was to copy binary gems and install them but there are still unsolved cross platform issues, I need to think about. Custom Icons ares even less likely to be done by me.

First, I’ll get the Linux and OSX “download if needed” working with an eye to allow the above crazy talk. Raisins Parity first.

[P.S] Since this is all pure Ruby now, Shoes 4 (jRuby) should be package up a shy into a 3.2 exe or osx tbz or a linux .run. That isn’t crazy talk. I don’t why they’d want to, but they could. Yes, I could also add an option to package for Shoes4 once they settle into a scheme that works for them.

Houston, we have download.

Things are looking up. (Test) Packaging for Windows can now trigger a download and install of Shoes before running the the script or shy. When I’m finished a Shoes person with a small script or Shoes app doesn’t have to deal with 7 to 15MB files to distribute. This much loved option of Shoes was broken many years ago. It’s about to work again. Then I hope to make it even better now that I know more and can cross compile all of the Windows C code with MinGW base compilers. I have a lot of testing and some minor bugs to fix but it’s looking very promising.

It’s been a long slog, deep into Windows do-do. I survived and did some good. It should be more fun soon.

Movin` on up

Moving on up the inverted pyramid to a penthouse in the sky (old TV show reference). Previously, I said the stub.c wasn’t Unicode friendly. I was wrong. It’s very much Unicode aware (in the Windows way). I can inject strings into shoes-stub.exe, and when the stub runs on Windows it can retrieve those strings.

Should I explain the significance? A pure Ruby script added binary resources to a Windows executable file (pe) and when that is run on Windows (and Wine) the stub.c/shoes-stub.exe can find them and use them. No winject.c or binject.c needed.

The inverted pyramid reference is all the code that has to be written or modified. At the moment, it doesn’t handle binary payloads – just strings. That’s a small matter of Ruby work. small being a code word for ‘more than I think’. Stub.c needs some work to move past Proof of Concept. Winject.rb need work. app_package.rb needs work and then add in the other 4 platforms. And then I do it again to implement my vision of all packaged Shoes (script or dir) would run a Shoe script to install the users script/app. But that’s all crazy talk for when the inverted pyramid becomes a diamond shape. Not there yet.

Download Shoes if Needed packaging is in sight. From any Shoes to Any Shoes. And the website to use is not hard coded in C or Objective C and can be changed with a knowledgeable user and a text editor. It might even play nice with Shoes 4. More crazy talk!

I think I can do it.

Which way is up?

Shoes 3.2 has met the future and the future is kicking my ass. I can see many reasons why the despised binject failed on a mingw stub and the winject (c) version. And the winject.rb – but I fixed that.

The exerb (pure Ruby) based winject.rb can create/inject string resources into the MinGW compiled stub.c -> shoes-stub.exe. It creates string table resources, but you have reference them with an id#, not a “string” lookup. So (per my previous post) “SHOES_FILE_NAME” is now Winject::EXE::SHOES_APP_NAMEy in Ruby and #define SHOES_APP_NAME 50 (from stub32.h) for C. The stub shoes-stub.exe will not look up by string name, it will use the number (50). That could be an annoyance
to future maintainers. Was it too hard to get exerb to create named string resources? Yes it was. It can create a StringTable for each injected String and that took major cowboy head scratching.

And now we meet the Windows Unicode monster. When MingW compiles a STRING_TABLE it produces a UTF16 bastard. Yeah, that might be controllable with some sort of #define but exerb doesn’t know about that (or handle it well). As it turns out, stub.c is not really unicode aware. Just enough syntactic sugar cover-ups to compile with MSVC 6 and MinGW.

I needed to fix stub.c to deal with the download site issue (it should be from a loaded resource, not a compiled constant). Bite the Windows Unicode bullet. I’d be lieing if I said that was going well. For now, I’m going to convert all UTF-16 resource strings to UTF-8 in stub.c with a char * shoes_str_load(HINSTANCE inst, UINT resnum).

It’s going to take some time.

Bootstrap to the future

[UPDATED – Sept, 22 2014 – changed Constants]

This rabbit hole is deep! And dark. I discovered winject.c can’t handle two string injects on a msvc compiled file (blank.exe). We already knew it can’t handle MingW compiled exe’s which is what led me into the rabbit hole.

I did manage to convince exerb to call my code for RT_STRING’s but I don’t know what I need to do to make it work properly and I can’t use winject,c for a base case anymore because I want something better.

Since no one cares about blank.exe and msvc, and I have to rewrite stub.c to handle which website and package (of many arch’s and versions) to download, I could define my own Shoes unique resources [user defined] and load them when the stub runs and write them from Shoes packager I also have to be aware of what Linux makeself can do. I want

  1. SHOES_APP_NAME – the name of the app
  2. SHOES_APP_CONTENT – the contents of the app’s shy
  3. SHOES_SYS_SETUP – optional, if exists it’s full Shoes exe to install first if Shoes isn’t already installed. Download and install Shoes if SHOES_SETUP doesn’t exist and Shoes doesn’t exist
  4. SHOES_SYS_SITE – the website (probably a cgi script) to download the closest Shoes3.2 from.
  5. SHOES_DOWNLOAD_SITE – website w/o http://, ie shoes.mvmanila.com
  6. SHOES_DOWNLOAD_PATH – path to cgi script ex: /public/select/win32.rb
  7. SHOES_VERSION_NEEDED – a gem like string filter (defaults to newest version avaliable. This is sent as part of SHOES_SYS_SITE request url. No guarnatee that the website will do what you ask

In my New World Order, SHOES_APP_CONTENT is always a a Gziped ( via RubyGems) tar ball. It will always be expanded in a temp directory. There will always be two files inside. There is the install.rb script in that temporary dir – Packager will create a simple one if there isn’t a custom install.rb specified when packaged by the user.

The simple install script is a Shoes app (possibly using the Shoes just intalled to run it). It’s job is to unzip/untar the second file (a .shy like thing) in the temp dir and copy it to the user’s choice of directory and run any OS specific things to create menus or shortcuts, symlinks, registery entries to the ‘copied’ app and other ‘stuff’. That installer could also check if there is a third file in the temp dir, call it ‘shoes-{app}-gems’ Another tarball to be unwrapped/installed by the Shoes-app installer (think Shoes.setup only smarter). It could contain the C compiled binary gems that the Shoes app needs but are not availale in the normal gems repositories. (ie, no compiler on the host). I’m not sure how well that gem stuff would work, but it’s early in the spec process and reality will bite back.

Yes, I am aware that evil can be inflicted this way. That’s also true for any Shoes app or ruby gem. There is a reason for tar balls inside tar balls. Linux makeself only allows one binary when packaging and its a hack for Shoes hack to find it. Two is out of the question. OSX may have a similar issue. Either one could derail my Windows plan.

Crawling on the hot coals

In the source code for Shoes 3.2.16 (the next one) (see github). I’ve retired binject and winject, the C code we all hate. In Shoes 3.2 it was only used to package for Windows (from anything running Shoes 3.2). I believe I can convince my copy of exerb-mingw/lib/exerb to do what (binject)winject did.

Is there a chance I’m on the wrong path and over committed by bias? Of course there is! Without a winject (C or pure ruby) and a shoes-stub.exe compiled with MinGW, there is no ‘going forward’ progress for packaging a Windows exe. blank.exe (from MSFT C) can not work – it’s hard coded to download from rin-shun/something/here and that site died a long time ago. I need to change the download URL inside it to my site (or one the user chooses). I need to inject a SHOES_URL (the variable site or exe at that site to download from). This is actually a big deal when it comes to time and details. It only sounds easy on paper.

Don’t tell Devyn

I stumbled across the exerb project which does many things I’m not that interested in but it does parse Windows pe files in pure Ruby and it can write itself to a new file and you can manipulate resources (.rsrc stuff) from Ruby. If it works, there is no need for winject (aka binject.EXE) ‘C’ code which would be a big happy dance in Shoes land – I’ll lead the parade. Even for experience C coders, (b)winject is dense and I still don’t understand where it goes wrong reading MinGW exe’s.

Does it work? It can copy the MinGW compiled shoe-stub.ex and the copied exe works as expected on Windows 7 and with Wine. That’s much better than the winject C code. I know I can add resource directory entries without the subordinate data and it will try to write that incompleteness to the final file. I have high hopes. I just need to learn about Window resource entries and I can inject SHOES_FILENAME and SHOES_PAYLOAD and SHOES_SETUP (if the user asks).

In case you’re wondering, Devyn was a young man on the Shoes mailing list a few years ago who believed Shoes and Ruby could write bootstrap itself and if only someone would do it then all the ‘C’ in the world could be replaced. Me, I’ve seen enough theories about Utopia (or the popular we’re all going to hell in hand basket, real soon now, we’re doomed, doomed). I challenge both these visions. Code (or facts) trump dreams and theories.

So, there is bit irony involved if I can get exerb to work with Shoes. I think I can and I’m going to use what works, theory or not.

NOTE: exerb only handles Little Endian. Since I only bother with Intel and Raspberry Pi’s arch, it’s not a problem for me. (I don’t think it is).

Winject – can’t find my way home

Ah, a Blind Faith reference. It occurred to me that the black hat crowd knows all about injecting resources into an exe. Even white hats do too if you google properly. Long ago I had to debug binutils/bfd. I don’t remember what the problem was or even what the goal was. It’s hasn’t gotten easier, but I’m pretty sure I can restore (b)winject to it’s former glory, only it’s work

I would not have written it the way _why did, but you gotta go with what you have and my way would not be a cake walk. If it could be gemified, it might be useful to others … Thats just crazy talk.

I devised a better test that should have occurred to me earlier. I want to compare what winject does without injecting any thing just load and save, for the msvc stub (blank.exe) and the mingw stube (shoes-setup.exe). I uncommented _why’s printf’s and added some of own. And I don’t want to point and click through Shoes. Or get the printf stuck into the Shoes Console (which can’t copied to the system clipboard).

There is tip here for Shoes developers.

copy.rb:

require 'winject'

def pkgtest teststub
  $stdout.puts "Test: #{teststub}"
  begin
    exe = Winject::EXE.new(File.join(DIR, "static", "stubs", teststub))
    exe.save(teststub.gsub("/\.exe/", "-copy.exe")) do |rlen|
      $stdout.puts "callback #{teststub} #{rlen}"
    end
  rescue StandardError => e
    $stderr.puts "OOPS: #{e}"
  end    
end

exe = ARGV[0]
pkgtest exe

See the problem? DIR is a Shoes constant, it’s not in regular Ruby. winject isn’t a gem, it’s an extension and it’s only inside the Shoes.

Then copy.sh. I probably should have named in test-copy or something.

#! /bin/bash
../../dist/shoes --ruby copy.rb blank.exe >blank-copy.log
../../dist/shoes --ruby copy.rb shoes-stub.exe >shoes-stub-copy.log

../../dist/shoes is where the Shoes exe is, relative to the two scripts. I’m calling Shoes which initializes it’s internal Ruby (and gems and extensions) and then runs the script – without bringing up the GUI so it’s fast. Under a second to run that test instead of minutes to point and click.

Shoes will do this trick behind your back when you try to install a gem that needs to be compiled. Should work on Windows too. Pretty handy.

COFF MINGW

Which do you want first, the good news or the bad news? Never mind, they’re identical. I managed to cross compile platform/msw/stub.c which ashbb and now I,named as shoes-stub.exe. Wine actually runs it and it displays a dialog that it is a shoes-stub with nothing in the payload. Exactly what static/stubs/blank.exe displays. Exactly what they should show if Winject hasn’t been used on them. The build instructions for blank.exe have been lost in time and I couldn’t get my mind and the existing rakefiles to co-operate on building shoes-setup.exe with MinGW so of course I did a little cowboy coding – just add the compile and link commands I want to a rakefiles task. No need for over engineering with fancy tasks and subtasks and blocks. No one going to run this task unless they’re trying to do what I’m doing, in which case there setup is going to very similar to mine.

I strongly suspected that blank.exe was compile by a MSFT compiler and thats why Ashbb went his special way with MinGW. It turns out stub.c is a C++ program. I consider C++ to be the spawn of good intentions from which evil arrives charring wonky MSFT C enhancements. Your opinion may differ. So I compiled it with C which winhttp.h wasn’t expecting and and and. It’s only got a couple of ‘probably harmless’ error messages and it does run. There nay be future trouble or work required to really get it correct.

I got rid of all the compiler warnings in winject. That’s nice even though they were all harmless.

What won’t work is Winject and a MinGW compiled stub. I suspected that was the reason ashbb went his direction. Confirmed. With my ‘new’ shoes-setup.ext, I get a segfault when winject tries to save it’s work. With gdb if needed. _why left comments in his ‘C’ code and some of them in winject provide a clue what he thought might be a limitation. I suspect I have found it too. _Why also left commented out printf’s so that’s good.

This could turn out to be a dead end – more work that I want to do or the details to troublesome to find. We need to remember this effort is only because I don’t like the current Shoes 3.2 packaging. It would still work again (as Shoes 3.2.15 does), since I haven’t deleted binject.

[update]
blank.exe has four (4) sections and shoes-stubs.exe has 8 sections. There are differences of course but nothing that is unreasonable. There are some Characteristics flags that might matter and they import dlls / symbols differently. Maybe. The icon section seems a bit confused. Winject (and binject) really only plays around with about the .rsrc section. winject.inject just adds an ruby array entry in to the adds field of the winject_exe_t struct (the field is a ruby array – _why did love to mix his C with rb_calls for expandable array and hash purposes). Sorting it out after all the injects from packaging calls is left to winject.exe.save.

I know that shoes-stub.exe is reasonable (from a Wine/Window POV). It’s only when winject gets involved that Windows does’nt like the exe. So, I need to package up just a script (not a downloaded shoes) with each of them and compare the .rsrc sections.

How deep is that swamp?

I’ve been researching the Windows problems with Shoes and packaging. I know, I just can’t leave it alone. A scab to be picked off before healing or some other sick metaphor.

In my opinion, the only Shoes packaging worth anything is the defunct “download shoes if needed”. Shoes 3.2.16 can pack a directory now (it creates a .shy, as 3.0, and 3.1 did) but running that critter on the destination is less than pleasing. Which is one reason all the Shoes folks gave up trying to fix it. However, some people can look into the abyss and pluck out the jewel. Is that me? I don’t know. I’ve given up before. I could do it again. I make no predictions on success.

Shoes 3.1 has a short circuit in packaging (download if needed) for Windows that only worked if you run Windows. I need to fix that so anyone can package ‘download if needed’ .exe’s At the moment, Binject (now known as Winject) has problems inserting (injecting) resources into a Mingw compiled exe. Works fine for a MSVC (6?) compiled exe. Everything Shoes touches or uses was compiled with MinGW. So I need to fix winject.c. And before that, I need to compile platform/msw/stub.c and with mingw and call it static/stubs/shoes-stub.exe and then I figure out what winject is doing wrong,

I could get really lucky or be in for a deep deep dive.