Microsoft Research has been making a bunch of cool data analysis-related launches at the upcoming Faculty Summit.
First, there’s The academic release of Dryad and DryadLINQ
Dryad is a high-performance, general-purpose, distributed-computing engine that simplifies the task of implementing distributed applications on clusters of computers running a Windows® operating system. DryadLINQ enables developers to implement Dryad applications in managed code by using an extended version of the LINQ programming model and API. The academic release of Dryad and DryadLINQ provides the software necessary to develop DryadLINQ applications and to run them on a Windows HPC Server 2008 cluster. The academic release includes documentation and code samples.
They also launched Project Trident , a workflow workbench, which is available for download:
Project Trident: A Scientific Workflow Workbench is a set of tools—based on the Windows Workflow Foundation—for creating and running data analysis workflows. It addresses scientists’ need for a flexible and powerful way to analyze large and diverse datasets, and share their results. Trident Management Studio provides graphical tools for running, managing, and sharing workflows. It manages the Trident Registry, schedules workflow jobs, and monitors local or remote workflow execution. For large data sets, Trident can run multiple workflows in parallel on a Windows HPC Server 2008 cluster. Trident provides a framework to add runtime services and comes with services such as provenance and workflow monitoring. The Trident security model supports users and roles that allows scientists to control access rights to their workflows.
GrayWulf builds on the work of Jim Gray, a Microsoft Research scientist and pioneer in database and transaction processing research. It also pays homage to Beowulf, the original computer cluster developed at NASA using “off-the-shelf” computer hardware.
Yesterday Dan, Pradeep and I presented “y!Vmail: voicemail for your Yahoo! Mail” at the Yahoo! University Hack Day Contest, winning the award for the 2nd best Hack! (jump to the demo video )
Our team with judges Paul Tarjan and Rasmus Lerdorf
The adventure started when I heard about Yahoo!‘s Hack U event:
Join Yahoo! web experts including Rasmus Lerdorf, the creator of PHP, for a week of learning, hacking and fun! You’ll hear interesting tech talks, hacking tips and lessons, and get hands-on coding workshops where you’ll work with cutting-edge technology. The week’s events will culminate with our University Hack Day competition—a day-long festival of coding, camaraderie, demos, awards, food, music and jollity (it’s a real word, look it up).
Years ago when I was in my teens, I was an avid participant on the school / college tech fest circuit. Almost every major institution in and around Delhi would organize annual technical festivals, hosting programming contests and software demo competitions. This was where I got a chance to showcase my creations and meet other hackers. Winning these events became a good way for me to pay off those telephone bills — web development in the dial-up age was an expensive hobby!
I decided to enter the Hack Day contest just for fun; it had been a while since I participated in one of these. It wasn’t about winning this time; I just wanted to do the whole “idea to execution to demo” thing with a group of friends, and spend hours screaming at each other over STUPID hard-to-find bugs that are actually staring at you in the face, high-fiving every hour as a feature milestone was scratched off the todo-list. The reward: to be able to stand in front of a group of people and say “Hey guys, look what I made!.” (If it’s hard to appreciate what this feels like, this video might help.)
Yahoo! gave away a bunch of t-shirts, this was on one of them
3 days before the Hack Day, I had an idea about building a phone-based interface for email. The idea was simple enough to build in a day, but fun enough to make an enjoyable demo. The only problem: I was already in the midst of a “hack” daymonth of my own; VLDB was due 3 hours before the start of the Hack Day, and I was already sacrificing sleep for LaTeX and Python for more than a week. There was no way I was going to be able to do this alone. Enter fellow grad students Dan and Pradeep. I told them about the contest and my idea. While they are both expert hackers, I totally forgot about the fact that people in Operating Systems research don’t really do a lot of Web Programming: “PHP….? I’ve never…” said Dan. I pointed them to the Yahoo Developer Network site and returned to my research paper writing madness. Hopefully by Friday evening, I would have a web-savvy hack team.
On Friday, I took a quick nap after my paper deadline, and walked over to the Hack Fest area to meet my team (who had become PHP and telephony wizards by now) and load up on caffeine and sugar that the Yahoo! folks had set up for us.
We split the work into two parts; Dan would build the phone interface while Pradeep and I would figure out the email and contacts API to write an email client backend. 7 hours later, we had the first version of our product up and running. We could call in and read emails. Happy with our progress, we decided that it would be wiser to go home and show up early next day. We ended up wasting a few hours the next morning worrying about the presentation: the lecture hall had spotty cellphone coverage, a deal-killer for a phone demo! Pradeep made a breakthrough here, discovering that an obscure panel on the wall was actually a secret speakerphone. Having resolved demo issues, we resumed coding and plugged in the remaining features: navigating through emails, email summarization, and email prioritization. The friendly timestamps feature (“4 minutes ago”) was stolen from my blog’s code (i.e. the Status header of this blog).
Around 3:30pm on Saturday, we updated our hackday entry:
by Arnab Nandi, Daniel Peek, Pradeep Padala
“Not everyone has a computer, but everyone has a phone.”
This hack allows people to access their Yahoo! mail through a 1-800 number, using ANY touch-tone phone.
Press 0 to open, * and # to navigate, 7 to delete. We figure out which emails are important, and read them first. We summarize long emails so that you dont have to listen to all of it. If you want to talk to the person, just press 5 — we’ll connect you.
APIs used: BBAuth, OpenMail, Contacts API, Term Extraction API
Hack presentations started at 4:00pm on Saturday. I started with a 20-second powerpoint pitch, followed by a rather entertaining demo. Using the lecture hall’s speakerphone we had the lecture hall call our service. Entering the correct PIN logged me in, which resulted in an entire roomful of people were now hearing the words “Welcome to y!Vmail. You have 5 new emails…”
Me pushing numbers on the phone
Here’s a short video walk through of our app:
More details at http://yvmail.info
A few minutes after the presentation ended, the prizes were announced. We ranked second. The winning hack was Brandon Kwaselow’s “Points of WOE”; a native iPhone app that allowed browsing and creation of placemarks on Yahoo! Maps. Congratulations, Brandon!
Overall, this was a very exciting and enjoyable event; I had a rocking good time hanging out with the Yahoo! folks and getting a cool project out the door with around 15 hours of work. I end with some lessons, acquired over years of doing demo contests:
Be creative, but avoid feature creep.
Split up into sub-teams, but make sure you’re pair programming most of the time.
Get Version 0 done Super Super Early. Then polish, polish, polish.
FINALLY finished the captcha.module for drupal. This is ONLY a first draft, lots of improvements to happen. Features:
* ability to protect any drupal form
* captcha API – make your own challenge response! (math and image are included in package)
Amazon Mechanical Turk provides a web services API for computers to integrate “artificial, artificial intelligence” directly into their processing by making requests of humans. Developers use the Amazon Mechanical Turk web services API to submit tasks to the Amazon Mechanical Turk web site, approve completed tasks, and incorporate the answers into their software applications. To the application, the transaction looks very much like any remote procedure call: the application sends the request, and the service returns the results. In reality, a network of humans fuels this artificial, artificial intelligence by coming to the web site, searching for and completing tasks, and receiving payment for their work.
It’s insanely ambitious, but I applaud the Amazon guys for coming up with something like this.
David Wouters at Upsaid.com has finally implemented the XML-RPC API for weblogging for the service. This means you can now update your Upsaid.com weblog using BlogBuddy, wBloggar, or any other weblog app. Cool!
I had started this XML-RPC project a few months ago, but was too lazy to actually go on to complete it. David then redid the whole thing himself, complete with weblogs.com ping and the works. If you don't know about Upsaid, I really sugest you go and have a look.
Finally had the Windows Programming Test today. Some blokes in our class kept postponing it for days, and it's really irritating because your schedule is all messed up because of that. Didn't do as well as I expected, but it was decent. It was good that we had the test - not because I wanted marks, but because atleast I sat and studied what the Windows API was all about. Having attempted at making my own Windowing library, getting to know what the real Windows is like is pretty cool.
My Windowing library - ah, it brings back old memories when I was in school. Sameer, my project partner used to sit around going, "naah - I think the class implementation was better if we did that". And I wrote the whole damn thing out. Helped me a lot - I must say. It helped me in get my 4GB hard disk - I have reason to believe the old one crashed because of the 100 reboots I used to subject my computer through under the name of "Video Memory Research". But the one thing I've learnt most from WinGUI is not the graphics, but about the sheer power of Open Source.