Archive for the ‘Programming’ category

Sending Binary Data with the Juno Framework

June 24th, 2010

Juno is a small Python framework similar to the Ruby Sinatra Framework. I needed a very simple way to look at a large set of png files in a specific way across a number of computers, so I opted to use Juno. It took me about twenty minutes to figure out how to send binary data dynamically. (That is, how to send image data via decorated function, not through Juno’s static directory convention.)

The following works, although it might not be the Juno developers preferred method.

def your_request_handler(web):
    content_type('image/png')
    append(string_of_binary_data)

It’s obviously simple but for me, it wasn’t obvious.

Email Yourself via HTTP-GET

May 14th, 2010

I am hoping to use Google AppEngine on a non-profit open-source project. To help me get familiar with AppEngine, I made a very small, simple webapp that sends a short message to any signed up users when they make a HTTP-GET request to their assigned URL. It’s useful if you have a very long running simulation or compilation that you would like to walk away from but be notified when it completes (or errs.) Assuming the email address used for your Google Account is also the one used on your smart phone, it basically mimics SMS without the headache.

The message should be shorter than 78 characters. Anything longer will be truncated. The message will be the email subject. To (feebly) prevent a quota DoS attack, each account is limited to 100 messages a day.

P.S. I learned only minutes ago that AppEngine apps cannot receive email from custom domain addresses, a feature I think my non-profit app will require. FAIL!

Fetishizing Programming Languages

May 1st, 2010

Learning a new programming language is fun. Different languages take different approaches to solving problems — they follow different paradigms. Learning a new language feels a lot like exploring, and I like exploring.

Exposure to more than one language is a good thing. However, once you are familiar with the broad paradigms, there exists a danger of conflating learning new languages with learning new ways to solve problems. In my case I forgot that programming languages are tools. Learning new languages can be seen as acquiring new tools. Learning a set of very similar languages is of limited utility because your not actually gaining anything new. There is almost no marginal benefit. To make things less abstract, compare the promisculous language learner with a carpenter. Carpenters don’t seem to suffer from the same error of judgement. There might be several brands of hammers offering slightly different features, but they all pound nails into wood.

I still enjoy learning new languages, but I don’t do so with such fervent desperation anymore. I’ve ceased my semi-conscious search for the best one. Instead, I finally realized my time is best spent studying algorithms, learning new programming techniques, and contributing to existing projects. This is much more rewarding than implementing the same software in twenty-three languages.

P.S. This post is a follow up to Consumerism and Programming Polyglots. I wrote Consumerism and Programming Polyglots a while ago. When I reread it today, I was very dissatisfied. It hinted at what I wanted to say, but it was inarticulate — mostly because I didn’t understand what mistake I was making by learning languages promiscuously. I think I get it now. Six month from now, I’m sure I’ll identify an entirely new aspect of my miseducation.

P.P.S. I wrote the title down, remembering someone had said it to me once when critiquing another post, Back to PHP. I searched my email before publishing this post and discovered it was Zed Shaw. This was disheartening. It took me several months since my original post to come to a conclusion that was accurately summed up by his flippant response: “All this fetishizing of technology is kind of pointless.”

JSON as a Universal DB Model DSL

December 30th, 2009

I write web applications rarely and reluctantly. I am not the most qualified to initiate a project based on the following ideas. If I did, it would likely  fall into a state of disrepair once my ephemeral attention has shifted back to playing with stock prices. However, I think it is a good idea. I am hoping someone who has both the time and interest codes it up. This is a new level of lazy.

var user_model = [
    // column name , generalized type,   , validation rule,   hidden?
    ['first_name'  , 'text'              , 'name_regex'              ],
    ['last_name'   , 'text'              , 'name_regex'              ],
    ['middle_name' , 'text'              , 'name_regex'              ],
    ['birth_date'  , 'date'              , 'over_18'                 ],
    ['location'    , ['double','double'] , undefined,        ,   true],
    ['addresses',  , 'has_many'          ,                           ]
];

Basically, from a Rails perspective, replace your ruby database migration DSL with code that parses the above JSON (e.g. user_model.json) and creates the appropriate schema. Replace your ActiveRecord validations with the validations from the json model. (Depending on your language, there may be differences in REGEX syntax. Having some well defined common REGEX definitions, such as EMAIL_VALIDATION, might be appropriate.) On the client side, you can display most forms automagically, based on the order defined in the json model (if it is in fact an array.) Furthermore, you can also run client side validations before submission. (If your validations are so weak that sharing them remotely represents a security risk, you are probably doing something wrong.) It might also remove the need for many controllers. A lot of CRUD is so generic that it might be possible to have a single /:model/:action/* controller. Scaffolding is not DRY. Finally, as the lingua franca, sharing model definitions outside your project’s language is simplified.

Any takers?

Consumerism and Programming Polyglots

October 28th, 2009

This is a follow up to Monday’s blog post, Back to PHP. In the scope of the project I need to develop, PHP is the best choice. My conclusion was correct. The reasoning I used to reach that conclusion was (mostly) incorrect. The post was basically a list of justifications that were a product of “Programming Polyglot Syndrome.” This post is a half-hearted retraction Monday’s post.

I dislike shopping for clothes. Unfortunately for me, social and professional conventions dictate that I must spend some time and energy dressing myself in something other than grey sweatpants and Birkenstocks. Shopping is a necessity and I shop hastily.

For example, I recently noticed my black dress shoes had a hole worn through the sole (oh, Aerosmith) last week. I headed to DSW. For me, the costs imposed by wearing cheap shoes are negligible. For most men, shoes are shoes. If they don’t look dirty, are properly shined, and are not 95% creased they look fine. Moreover, I care more about my checkbook than my feet (the later are mostly self-repairing, the former is not). Once I find a pair of shoes that are not pink, are not expensive, and are not four sizes too large, I am done. I am satisficing.

People who enjoy shopping are not satisficing. That frustrated me until this morning. An introspective friend of mine was describing why she enjoyed shopping so much. She was able to recognize that, for her, shopping is about buying possibilities. She doesn’t shop for new tops. She shops for the experiences she believes – rationally or irrationally – the top will elicit: the attention of some man she finds attractive or the jealousy of some woman she dislikes.* She is buying hope.

Now, because this conversation was taking place via GChat, I was doing other things at the time. At one point, I checked the traffic stats for the unexpectedly popular post I wrote on Monday, Back to PHP. In the Back to PHP post, I tried to justify switching back to PHP, a language with a lot of warts. Thinking about it in the context of my friend’s consumerism, I noticed that it’s roughly the same thing. I have a project that I have no strong interest in but I must write. By switching to PHP I was trying to optimize away most uninteresting parts of development – mainly deployment. Cognitive quirks lead me to half-conclude that PHP would make the development go swimmingly well, with no bugs, deployment issues, or mental energy required. Magic!

This is not the first time I have learned or relearned a language because I fostered the delusion that it would {motivate me; dissolve all complexity; …, cure cancer; bring about world peace}. Actually, for me, this is often the primary impetus. In most cases, given the context of the problem I actually set out to solve, this turns out to poor investment of my time.

That is not to say that learning many programming languages is a bad thing. Different languages have different conventions, paradigms, common practices, etc. Learning many languages and paradigms pays a dividend. You become a better programmer and you carry new ideas across language lines. There is nothing wrong with learning for the sake of learning. However, if you are learning something new to avoid doing something you don’t want to do, you are probably not going to make any gains in productivity for that project. You are just procrastinating.

In the case of my project, PHP is the best choice because initial deployment in environments that I do not control is one of the most important considerations. PHP is very simple to deploy initially and is very common. However, most of the justifications I provided were not related to this issue. I just wanted to (re)learn a language.

Although, to be fair, Factor might be my best bet. Forget all this introspection!

* This is actually pretty interesting if you think about it. “Peacockery” in fashion is more apropos than realized. Not only do people wear things that are ostentatious to attract attention, but doing so becomes a requirement for those who otherwise would be happy in sweat pants. Adopting fashion trends is less about conformity than it is about leveling the playing field for attention. It may be less about showing fitness with clothing acting as a proxy for wealth and more about Fisherian runaway. F*cking, peacocks.

P.S. “Programming Polyglot Syndrome” is almost an extension of NIH. Coding something from scratch can be more interesting than learning to properly deploy and use someone else’s code.

Going Back to PHP

October 26th, 2009

I consider myself a programming polyglot. I am proficient in five to seven languages and am competent in many more. Consequently, like many programming polyglots, I am a bit of a snob. PHP programmers tend to be unilingual or bilingual (Javascript). They are unenlightened. If you asked the average PHP programmer to explain a closure, they would probably express bewilderment. Brutes! Like I said, I am a bit of a snob (and the preceding text was satirical…mostly).

As is the case with much snobbery, this is ill-advised. Prior to PHP, I remember writing applications through CGI, mod_perl, or full-blow C extensions to Apache httpd. It was not very productive. Enter, PHP. From the beginning, PHP was designed as a template language to make web development easy. As such, additions to the language and its tool set were evaluated against a fitness function that emphasized ease of use and productivity for web development. At the time, it was the strongest solution to a problem of increasing prevalence.

Fast forward a few years. I had been out of the web development scene completely when a bout of temporary insanity inspired me to attempt building an online dating site (thanks, Markus Frind). Since I used Ruby extensively for non-webdev, Rails was my obvious framework choice.

Does Rails have advantages that cannot be replicated in PHP? Of course! The most glaring advantage of Rails over PHP is the fact that PHP is not Ruby. Ruby is lovely language. Seriously. Ruby allows you to think in code. It is incredibly productive. You can take very complex abstractions and implement them in Ruby with ease. However, web development is not very complex. Most web development is CRUD. Sure, Ruby might be slow (although 1.9 is looking nice) but most applications really do not require high performance. And when scalability does become an issue, Rails is rarely the bottleneck.

Do I miss ruby gems? Sure, but Pear works. Moreover, integration with PHP is often easier than it is with rails. When was the last time you noticed that popular api or library lacked a PHP interface? Most sites release official PHP libraries (s3, ec2, facebook, gdata, …)

Conversely, PHP has one several advantages over Rails. Most notably, immediacy. Deployment via SCP is very satisfying. Moreover, PHP now has many rapid-development frameworks that employ MVC and active record patterns (e.g. CakePHP). While you don’t have the elegant sexiness of Ruby, you have almost everything else.

Do PHP programmers tend to be weak programmers who produce mediocre code? Maybe. If so, it can probably be attributed to the law of large numbers. There is a virtual army of PHP programmers; there are very few DHHs. For large-scale, very complex problems it may be better to use a language that naturally excludes poor to mediocre programmers, but this is not true for web development. Besides, there is no shortage of criticisms of the rails community.

Finally, there is nothing to stop you from using PHP for the CRUD front-end and Ruby or some other language to do more complex heavy lifting (e.g. spidering, categorization, etc). It seems to have worked well for facebook.

echo "Hello, Old Friend";

UPDATE(OCT-27-2009 @ 1:39)

I mentioned below and repeated in the comments section that I have used Capistrano (a site which, given the current context, ironically uses PHP) and Passenger, but, for reasons I could not qualify, I thought PHP was simpler. Mark just helped finalize the thought for me: PHP is usually preinstalled and ready for rapid deployment — i.e. immediate initial deployment. Heroku is wonderful, but not every hosting service is Heroku. If you want an application with diverse end-users, PHP is more likely to be eligible for simplistic initial deployment. It may seem like a minor issue, but in my experience, it is not.

END_UPDATE
This post was motivated by an open-source project I am currently coding. It is not yet released but for the purposes of this footnote, pretend it will have a deployment pattern similar to Wordpress. I started writing it in Rails. However, after speaking with several people who will be end-users, I realized that the most important factor would be ease of deployment. While Phusion is a game-changer for deploying Rails applications, I think PHP is still easier for most web developers.

Also, for the sake of disclosure, I should make it clear that I am not an expert web developer. As you might notice from this blog, most of my work is in complex systems. Most of my time is spent writing code in C or C++ with Ruby acting as glue.

Warning: NVIDIA CUDA memset bug

May 9th, 2009

Calling cudaMemset() on some platforms does nothing. It took me two hours to figure out why the result of my computation was so bizarre. My experience was similar to many people who posted on this informative thread. In emulator mode, the function performs as expected; When running on the device, the memory is not set.

CUDA for Financial Modeling

December 30th, 2008

Kudos to NVidia for CUDA; It’s fast — really fast.

CUDA allows you to harness the parrellel power of a CUDA capable GPU for (semi-) general computation. The result? Hundreds of cores running your binomial option pricing models, Monte-Carlo simulations, or Black-Scholes computations.

I started learning CUDA yesterday; I wrote my first simple CUDA program today. The library does have a non-negligible learning curve, but it is not steep. It largely is a matter of learning the most efficient ways to work with CUDA (e.g. shared, local, or constant memory). Happily, this is an incremental process; You can learn to write bad yet working CUDA applications while slowly learn to write them better; And, as a bonus, even your bad code is likely to run laps around your CPU (for finance apps anyway).

I am currently writing a Poker hand simulator (to compare to the OpenHoldem’s speed) but, once I am comfortable with the library, I will be porting my c++ option pricing algorithm. With CUDA, my algorithm will now be (closer to) real-time!

P.S. I bought the GForce 9600GSO which was only $99 at Best Buy; This low-end NVidia card achieved the following results for the binomialOptions.exe sample included in the SDK:

Using single precision...
Using device 0: GeForce 9600 GSO
Generating input data...
Running GPU binomial tree...
Options count            : 512
Time steps               : 2048
binomialOptionsGPU() time: 92.526009 msec
Options per second       : 5533.579236
Running CPU binomial tree...
Comparing the results...
GPU binomial vs. Black-Scholes
L1 norm: 1.484960E-004
CPU binomial vs. Black-Scholes
L1 norm: 1.045247E-004
CPU binomial vs. GPU binomial
L1 norm: 4.464579E-005
TEST PASSED
Shutting down...

Not bad, Eh?

The Every Pattern Ever Engine

August 28th, 2008

Around my sophomore year of college, I wrote an application to analyze technical analysis trading patterns. I read a few books on technical analysis and thought they were horribly unsophisticated. For the most part I found that technical analysis books were basically if match(patternX) then take(positionY).

I had not yet reached any mathematical sophistication, so I did not liken chart pattern trading to tea leaf reading. I did, however, wonder why they relied on this one set of chart patterns if others, especially those not published, might exist.

To satisfy my curiosity, I built an application that constructed every pattern that ever occurred in the listed SP500 stocks since 1980. It was actually fun to write and remarkably fast after all the graph indexes were created. I could choose to set how many days would define a pattern, how close the movements would need to be to considered a match, the importance of relative volumes, and countless other measure that I have long since forgotten.

Returns were based on either holding for a static period length; holding until history suggested deviation from the past pattern and all newly matching patterns suggested movement against my position; and other plays that I can no longer remember. After all matches had been assembled and all returns were generated, I could browse the resulting return distributions for each unique pattern. The result: many appeared to be very profitable but the overall return out of sample was close enough to zero to be discouraging (I am purposely ignoring my current knowledge of statistics for this presentation).

I then went a bit further and looked up trading patterns mentioned in all the fantastic technical analysis books. It was clear that they were no more likely to be included in the strong, in-sample returns set than any random pattern. My intuition was correct; Technical analysis was unsophisticated and, simply, tea leaf reading.

Granted, the experience was incredibly useful and worthwhile. I would not discourage anyone from repeating my experiment; You will learn A LOT. It helps build an intuition for statistical inference prior to studying statistical inference — although studying statistical inference first is not a bad idea.

Many of my subsequent experiments were heavily shaded by this early experiment; Many posts will echo lessons learned.