Oleg Andreev
Month
This is an evolving document describing sRuby, a subset of Ruby that can be easily compiled to fast low-level code. The purpose of developing sRuby is to use it to implement a Ruby virtual machine. However, we anticipate that it can be used to write Ruby extensions that needs to bridge the gap between Ruby and a low-level language © in an easy and portable way.
Robert Feldt
April 4, 2001
Suppose you have that kind of user interface:
The efficient rendering algorithm would be the following:
1. Cell should render its content manually: that is not using multiple views, but using a single contentView which is updated programmatically in drawInRect: method.
2. Each time cell is rendered, it checks the cache of downloaded images. If image is not present, cell schedules request to download it. Cell should remember the request URL to be notified when data is ready.
3. Download request goes to FILO queue with async NSURLConnection delegate. Async API uses the main thread which is notified when data is ready.
4. The queue size should be limited by a maximum number of visible cells. The most recently requested images go to the end of the queue, while the head of the queue is truncated each time new request is added. This way scheduled requests are removed when corresponding cells become invisible.
5. Download callback should lookup [tableView visibleCells] to find the exact cell which requests the downloaded image. The cell could be different than the one started the request (remember cell recycling!). You cannot just call [tableView reloadData]: it works poorly when a lot of images are loaded while you scroll.
6. In addition to downloaded images cache, there should be pre-rendered images cache: each time cell draws itself and finds the downloaded image, it should rescale and crop the image. The rescaled image should be put in the cache, otherwise scrolling won’t be smooth. Of course, if the downloaded image is already sized as needed, this step could be omitted. See UIGraphicsBeginImageContext.
Toolkit summary:
— download requests queue with async i/o (no NSOperations, no background threads)
— manual cell rendering (no subviews)
— individual cell update (do not reloadData)
— pre-rendered images cache
— discard pending downloads for invisible cells
“Tracing and reference counting are uniformly viewed as being fundamentally different approaches to garbage collection that possess very distinct performance properties. We have implemented high-performance collectors of both types, and in the process observed that the more we optimized them, the more similarly they behaved — that they seem to share some deep structure.” David F. Bacon at al.
Great article explaining multithreading issues on multiprocessor systems.
Reference-counting is traditionally considered unsuitable for multi-processor systems. According to conventional wisdom, the update of reference slots and reference-counts requires atomic or synchronized operations. In this work we demonstrate this is not the case by presenting a novel reference-counting algorithm suitable for a multi-processor system that does not require any syn- chronized operation in its write barrier (not even a compare-and-swap type of synchronization). A second novelty of this algorithm is that it allows eliminating a large fraction of the reference- count updates, thus, drastically reducing the reference-counting traditional overhead. This paper includes a full proof of the algorithm showing that it is safe (does not reclaim live objects) and live (eventually reclaims all unreachable objects).
1. Some GCs do pointer recognition in the arbitrary data array (e.g. Boehm-Demers-Weiser GC); this is not necessary if GC should track objects of the known structure (e.g. Steve Dekorte’s GC)
2. If we track object references only, there’s no need to fight fragmentation of the GC-managed heap: all entries are of the same size.
3. After coding Obj-C for a while, I’ve noticed that the only issue which should be resolved by some kind of garbage collector is circular references. Retained properties and autorelease pools already help to avoid manual retain/release calls. That is: allocation should be always succeeded by autorelease and all the properties should be nullified on deallocation (this could be done automatically).
I wonder if it is possible to use a simple reference-counting mechanism with a simple referential cycles resolution, thinking that way we can imagine a very simple and efficient garbage collector.
References:
1. Minimizing Reference Count Updating with Deferred and
Anchored Pointers for Functional Data Structures by Henry Baker
2. Concurrent Cycle Collection in Reference Counted Systems by David F. Bacon and V.T. Rajan
1. When using sandbox user account, do not sign in using Preferences, just sign out and sign in when your app asks so.
2. Transaction receipt should be Base64 encoded before sending to Apple verification service. This is not mentioned in the docs.
3. Subscriptions should not be restored using restore API. Do not even try: Apple suggests sending relevant data from your application server.
4. Title/description you set in iTunes connect do not really matter if you supply products from the server. Only product id and the price tier matter. Everything else you supply by yourself.
5. Do not forget to sort the products elsewhere since it is impossible to sort them on iTunes Connect site.
6. Transaction receipt should be verified at apple server for two reasons: 1) secure validation; 2) this is the only way to get transaction details
“That is what happens when a conventional generational garbage collector is used for a program whose object lifetimes resemble the radioactive decay model. The collector simply assumes that young objects have a shorter life expectancy than old objects, and concentrates its effort on collecting the generations that contain the most recently allocated objects. In the radioactive decay model, these are the objects that have had the least amount of time in which to decay, so the generations in which they reside contain an unusually low percentage of garbage. For the radioactive decay model, therefore, a conventional generational collector will perform worse than a similar non-generational collector.”
Paper by William D. Clinger and Lars T. Hansen (PostScript)
The major feature of a dynamic language is interactivity. With Smalltalk you may run the program and inspect/change it at runtime. This implies some GUI for VM with built-in crappy text editor: you don’t edit files, you edit objects now.
This does not sound very comfortable for many reasons. First, you would always want to have a “canonical” state of your application which is not affected by runtime mutations: that is, plain text files stored under some version control. Next, you would like to use a different text editor or GUI and it is much simpler to achieve when you operate with plain files instead of fancy VM/language-specific API.
How do we combine interactivity of Smalltalk with text file editing? Let’s take the puriest OO language ever designed: Io.
1. Each file contains an expression.
2. The only way to load the file is to evaluate it in context of some object: object doFile("file.io"). The return value would be a result of the expression in the file.
3. We may have a convention that some files return a prototype object: the object which is used as a prototype for other objects created in runtime.
4. To load “prototype object” we use a special loader object which would track the file-to-object mapping: Article := Loader load("article.io")
5. Loader monitors the filesystem and when some file is changed, it loads it into another object and replaces the prototype with that object: Article become(load("article.io"))
6. At that point, all articles in the system suddenly have another version of Article proto.
You have to follow some safety rules. For instance, proto’s descendants should not modify the proto and rely on such modifications.
Of course, this method still does not allow you to change/inspect any object in the system. For this to work you may put a breakpoint message somewhere and use a debugger after the proto is reloaded and VM stepped on that message. Or wire some Smalltalk-like GUI to your app.
Simple proto-based reloading helps development a lot and in contrast to class loading methods with full app reload, works faster and for full range of source code including all libraries. Rails dependency system does not reload gems, but does a pretty good job with constant reloading. All ruby/rails issues with global data applied.
Now, working with Objective-C where nil eats messages, I realized that the code is more elegant, but it takes significant amount of time to debug it. You create if/else branches and breakpoints to trace the nil, then you fix the bug which causes it and erase the conditional code. You get your elegant code back and wait for another issue to arise later.
“Essentially, Haas Unica came about as a result of analysing the original version of Helvetica, its variants (as they were in 1980) and similar faces and seeking to improve them - to produce the ultimate archetypal sans serif face. A single face to unite them all, if you like. ”
See also: From Helvetica to Haas Unica (flickr set)
The paper discusses how thread-oriented programming is more efficient (in terms of performance and development cost) than event-oriented.
My personal observation is that cooperative multitasking (based on coroutines, fibers) requires less and easier to read code comparing to evented callback-based code.
The Objective C syntax is poisoned with nested square brackets:
[[[Class alloc] initWithApple:a andOrange:o] autorelease];
First, lets move opening bracket behind the name of the receiver:
Class[alloc][initWithApple:a andOrange:o][autorelease];
You may agree that this is much easier to write now. However, at this point we lose compatibility with ANSI C (think buffer[index]).
Lets omit brackets for messages without arguments and use a space as a delimiter:
Class alloc [initWithApple:a andOrange:o] autorelease;
At this point we may get back compatibility with ANSI C by making a non-context free grammar (parser should recognize that a[b:c] could not be used for index operations).
You can implement exactly that syntax in Io using the standard language features.
Stylesheet and javascript URLs and content should be controlled by application code. Putting static files into public folder is so nineties.
Before starting a work on a distinct feature, you create a branch:
$ git checkout -b myfeature
You write code, create fast commits, merge in master, rewrite code etc.
$ git checkout master$ git merge myfeature --squash
Now you have merged all the changes into the working tree, but not committed in the master branch (because of --squash option)
You may git add some files to produce nice commits as described in the previous article.
These rules are designed for an easy code review using “git log -p”. This command shows the history of commits with patches.
1. Commit message should include task reference number (# of ticket/case in bug tracker, url of wiki etc.). If there’s no reference number, then the ticket must be really trivial or include refactoring only.
2. Commit represents an atomic working patch. No “WIP” commits with undefined behavior are allowed. In your private branches you can do whatever you want, but when merging to master, you must aggregate commits in a set of working patches. If you don’t do that, the single feature would be spread among 30 commits with arbitrary code being written and erased between the start and the end.
3. Commit should be small. You should split a big commit in a few independent ones. More safe commits should be stored first. Good example: you had fixed some performance issue. First, commit a benchmark which shows the previous performance, then commit an updated code. This helps to test the previous code using newer benchmark without manipulating code by hand.
Rule 2 tells you not to pollute master branch with tons of WIP commits and rule 3 tells you to squash WIP commits wisely: do not put everything in a huge patch.
It is much easier to follow these rules when you look what others do with the code using git log each time you pull updates.
“There are two basic type of method: ones that return an object other than self, and ones that cause an effect. […]
As a general philosophy, it’s better to try and make your methods one type or the other. Try to avoid methods that both do something and return a meaningful object. Sometimes it will be unavoidable, but less often than you might expect.”
“But Apple require that this app be paid, not free, in order for us to offer In App Purchase. So lets look at that again, the same user downloads the app for $0.99 assuming it’s a one time payment, then launches the app to find that he only gets 30 days of service for the $0.99 he just paid. Furious he leave one star reviews all over the place even though we went to great lengths in the iTunes description to spell out the exact nature of the subscription and costs (but no one actually ever reads that stuff).”
While searching for “tell, don’t ask” I have got an interesting wikipedia article.
Is there a CS book teaching us how to write big complex programs?
- components identified by URI (“RESTful partials”)
- precise invalidation on data update (no timeout-based silliness)
- easy to extend, test and debug
The biggest advantage of dynamic languages is interactivity. With dynamic language you can open any part of the running system, change something and see how it behaves under these particular conditions, immediately. This dramatically improves design cycles, completely eliminates compile lags and helps to debug efficiently.
Smalltalk/Self guys got it more than 30 years ago.
It is pity to see how current Ruby/Python/JavaScript/etc. frameworks are less interactive than C++/Java within some modern IDE (like Visual Studio).
If the dynamic VM is a move forward, then next step are highly interactive tools. Everything else is just the same old story.
See also real life benefits of dynamic languages at stackoverflow.
Based on hash table vs. message-receivers and activatable slot, not value.
1. Every slot is activated on direct access. Non-activatable slot access raises exception.
2. x := y creates getter method(getSlot(“_x”)) and setter method(v, setSlot(“_x”, v); v).
3. x = y is parsed as x=(y) (i.e. message x= with argument y).
4. No ::=operator.
5. Method definition: obj setSlot(“add”, method(x, self + x))
5.1. Method definition macro: obj def add(x, self + x) (could be implemented in Io itself)
Pros:
- cleaner setters and hooks for setters;
- smaller syntax;
- uniform message dispatch: each message is processed by a method;
- safety: no need to use getSlot(“x”) for method arguments when activatable value could be passed (relevant to any abstract algorithms).
Cons:
- performance hit since local variable access should perform double hash table lookup; this could be optimized by storing hidden variables (_x) in a plain array.
What do you think?
1. Use divs with float:left/position:absolute and negative margins ONLY for the global page layout.
2. For inner modular things like “thumbnail with centered image and centered caption” NEVER EVER use layout tricks mentioned above. Always make sure the module does not require specific outer tags and styles. This is generally possible using tables.
Reasoning: when the smart object is inserted into unprepared environment (div or table) it is nearly impossible to put it into correct position since it has lost its height which should stretch outer container.
This is an important addition to the previous article.
Yesterday I have stated that every incremental development process suffers from increasing module coupling by definition. Smaller steps give you flexibility to turn around a current point in the development process, but not to jump out of it.
In previous article I have completely missed the first statement and started talking about “refactoring 2.0”. In fact, when you have reached first N lines of code in your project you should start a new feature from scratch (literally: create new folder, git init, etc.) This action could be considered as a small jump out of the current environment towards the latest requirements.
When you start building something side by side with the existing environment, you are forced to define some minimal API for the existing code to communicate with the new feature. This could be object-oriented API, config file or network protocol. Maybe you would need to refactor existing code in order to provide such API. In result you would produce two less coupled modules which will give you more flexibility as project gets bigger.
An observation: smaller module is easier to fit into a reasonable API. Complexity grows exponentially in respect to code size.
2. Immutable state is something I don’t care for, so it worries me that referring to map, filter, etc as “functional programming” may give people the impression that they have to swallow this immutable state business in order to use these things.
The Danger of Equating Map and Filter with Functional Programming
Inspired by You Can’t Get There From Here c2.com article
Every incremental development process suffers from increasing module coupling by definition. Smaller steps give you flexibility to turn around a current point in a development process, but not to jump out of it. With incremental process you are reaching local optimum: the best solution for the problem you are not solving today. But this is not the real issue (at least, you can sell it to someone else). The issue is that you can’t move incrementally from the local optimum due to high coupling. The only way out is to take independent components which are suitable for the new task, jump out of the current point and set up new process based on these components. Efficiency of this jump is measured in total relevance of all these components.
In other words, we need some insurance that some critical amount of investment (1 month, $100K etc.) is not thrown away as a whole thing. To achieve this we should keep the work splitted into small distinct pieces, each of the acceptably low cost.
It is usually recommended to refactor the code in order to extract abstract entities and generalize their API. However, it looks like a stupid game in the same playground: a single project directory tree with 1000 files in it.
Lets take a look at search tree balancing principle: each node should have some optimal number of children. If it has too many children, we have to evaluate linear search in the node. If it has too few, we have to evaluate linear search through the linked list instead of a tree.
Our asset is the code. The efficient evaluation of the code requires to keep it in a good shape. This could mean the following:
- N lines of code per method
- M public methods per class/module (+ M private)
- F modules/files per folder
- L levels of folders per library/dependency
- D libraries/dependency per product/another library.
Each figure is average. You can have 10*N lines method as long as there are ten N/10 line methods. The ultimate goal is to have maximum L*F*M*N lines of code per program (as well as M*N lines per class).
Figures could be something like that: N=7, M=7, F=17, L=3, D=7.
The idea is to limit the amount of code you work with. If you do so you would be pushed to extract least coupled parts out of the project, therefore making them more valuable individually and giving more focus to the essentials.
This implies slightly different mindset comparing to traditional refactoring. You do not look for a way to restructure the program just for making it cleaner: you look for a way to keep as little code as possible by extracting least relevant code into separate external modules.
“Another example of the inefficiency of large organizations. Individuals have little to gain in successfull projects (the company won’t make them rich), but much to lose in unsuccessful ones (they could loose their job). So the rational decision is to avoid risk, as it is not balanced by return.”
See also Python paradox
“It looks like what they’re doing are stackless user level threads and this means they don’t play nice with C calls as they don’t allow calling into C and then back into the language (as they can’t save the C stack). This may not sound like a problem until one considers how almost all C library bindings involve callbacks (xml parsers, graphics, audio, media processing, networking, etc).”
Caution: the post is 3 years old.
The most interesting one is ooc — another attempt to add objects, inheritance and improve packaging in C. via Steve Dekorte“An event to bring together bright folks working on unfinished or recently finished programming languages. Even relatively young languages like Scala would not qualify; this event is all about the sharpest part of the cutting edge.”
We assume you had Leopard with standard Ruby shipped with OS, tons of macports and rubygems already installed. Then you install Snow Leopard on top of it (not clean install).
The problem is that standard 10.6 dynamic libraries all went 64 bit and could not be linked with 32 bit code. This could be fixed by rebuilding/reinstalling all the macports and rubygems.
1. Install the latest Xcode shipped with Snow Leopard.
2. “port” command will fail with a message about incompatible Tcl architecture. The proper version of macport is 1.8 which is not released yet. You can obtain it from SVN trunk and built it by hand (./configure && make && sudo make install). I have also added trunk version of ports to sources.conf (see the link above for instructions) to be sure that I have the latest SL-compatible ports in the list. Maybe this is was wrong assumption, but it worked for me just fine.
3. Remove all ports:$ sudo port -f uninstall installed
4. Update rubygems to 1.3.1 at least (please google for instructions).
5. Remove vendor gems (gem uninstall refuses to remove them and fails to do batch remove):
$ gem list | cut -d" " -f1 > installed_gems
$ sudo mv /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8 /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8.bak
$ sudo mkdir /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8
Note: installed_gems file contains a list of all installed gems so that you can cat and xarg it for installing all gems back.
6. Uninstall all gems
$ sudo gem list | cut -d" " -f1 | xargs sudo gem uninstall -aIx
7. Make the rubygems use 64 bit architecture.
$ cat installed_gems | xargs sudo env ARCHFLAGS="-Os -arch x86_64 -fno-common" gem install --no-ri --no-rdoc
Note: it is NOT i686 (as I thought it should be), it is x86_64 instead.
Here are convenient aliases for “sudo gem install” for both architectures:
alias sgi32="sudo env ARCHFLAGS=\"-Os -arch i386 -fno-common\" gem install --no-ri --no-rdoc"alias sgi64="sudo env ARCHFLAGS=\"-Os -arch x86_64 -fno-common\" gem install --no-ri --no-rdoc"
You may also put alias sgi="sgi64" in your .bash_login on snow leopard.
Now we all can proceed doing productive work. \o/
References:
1. http://www.nabble.com/-MacPorts—19446:-openssl-fails-to-compile-on-x86_64-td23247203.html
2. http://jaredonline.posterous.com/got-mysql-to-work-with-rails-in-mac-os-106-sn
3. http://cho.hapgoods.com/wordpress/?p=158
“Rule 1. The sum of a signed value and an unsigned value of the same size is an unsigned value.”
“If developers are allowed to write "to the hardware,” the result is a broken platform where the vendor can’t move forward without breaking the apps.“