Suman Srinivasan | CS Research @ IRT

Round trip time measurements using ICMP and TCP

2009-08-14T14:30:00.004-04:00

ping is a great tool for measuring round-trip time delays, but:

Sometimes, ICMP is not enabled on the end hosts (increasingly now)
ICMP RTTs may not provide the right delay metric for real-world applications.

I searched and found two good "TCP Ping" tools, which basically seem to create a TCP SYN message and check the time delay in getting a SYN-ACK response from the server.

This should be a more realistic measure of RTT for real-world TCP traffic and should be possible to do on any public node.

Here are two that I found to be helpful

TCP Ping: Download script here and more details here (requires tcptraceroute)
Scapy TCP Ping (using Python): Doesn't seem to actually do RTT measurements... (requires python and scapy library)

UPDATE: scapy can do round-trip measurements itself. You need to scapy.py file and some basic tutorials...

Sending customized TCP packets using jpcap and libpcap

2009-07-15T16:33:00.002-04:00

Sorry I haven't blogged for a while; I've moved to Twitter.com/sumanrs for most updates. I'm only using this blog for longer posts now.

So my task this week was to create customized TCP packets for the NetSerV experiment. I would have had to look for a user-level TCP stack ... or use the popular pcap implementation, which can generate packets on the network.

And since this has to run on Java, we have to use jpcap (there is an older version on Sourceforge, which I did not try.)

I was able to successfully create a program that can send TCP packets, both on Windows and Linux. And was able to compile and run SendTCP.java, which can send packets

To get the TCP packets to send properly in a local network and be received at the remote node, you need to know a few things:

Make sure to correctly specify source and destination MAC addresses. You can get the source MAC address by doing:
ether.src_mac = device.mac_address;
You will have to manually set the receiver's MAC address, or look at ARP.java to figure how to do it over the network.
Make sure to correctly specify source and destination IP address. Can be done through InetAddress.getByName() as in the example.
For unknown TCP ports that don't have any daemons running on them, you may not get a ACK response for a SYN packet, though you will get a TCP RST packet.
If you send it to a port that has a daemon, you will get a nice ACK packet, which you can handle properly.

Windows ran succesfully, but I learnt a few things for the Linux install since I had to run it on a AMD 64-bit processor:

There seems to be no version of jpcap download on the jpcap site. So you will have to build it yourself.
Download the source from the jpcap site.
Follow instructions for source install here.
You need libpcap and libpcap-dev installed. libpcap is most likely already installed; if you want libpcap-dev, do "sudo apt-get install libpcap-dev"
You can install the native library and the JAR by following the instructions in #3.

Spring 2009: NetServ + candidacy + TA

2009-01-26T15:03:00.003-05:00

As the Spring 2009 semester begins, I am hitting the ground running for my candidacy exam in late Feb. My topic is "Ubiquitous Communication Models for Different Network Scenarios."

While at it, I am also reading up on the following topics for the NetServ project, which I will soon start work on:

And, finally, I am TAing the COMS6125 Web Enhanced Information Management again this semester.

The Beginnings of Spring 2009

2009-01-07T11:53:00.001-05:00

Back to work for Spring 2009. I am now done with courses - its a wonderful feeling to not have to be aware of when the semester starts and ends. :D Well, except I am TAing Web-Enhanced Information Management again this Spring semester... and am looking forward to it.

The downside is: I do start working on research two weeks before school starts ... but that's OK.

Right now, two items require my fullest attention:

Preparing my presentation for the IEEE CCNC research conference on the BonAHA framework (this weekend!)
Finalizing the list of papers for my candidacy exam (which will be in mid-Feb? I hope?)

Parallel computing and communication/network models - the "parallels"

2008-11-07T12:25:00.003-05:00

Apologies for the pun, but one of my readings over the last week has been to read up on parallel computing in order to find some parallels between the opportunistic applications I have been working on (such as 7DS) and parallel computing.

Even though the two topics look different, there are actually several points in common.

Below is a list of topics, extracted from the textbook Parallel & Distributed Computing Handbook by Albert H. Zomaya, published by McGraw-Hill.

PRAM (Parallel RAM) model; Methods for distributing and parallelizing algorithms
Failures in parallel systems (Byzantine, other)
Dataflow models
Checkpointing (for recovery)
Distributed software systems (this is similar to RMI)
Data structures for parallel processing
Shared memory, memory consistency

More details to come soon.

Finished BibTex on IRT publications page

2008-10-22T11:40:00.001-04:00

Today I fixed up the BibTeX links on the IRT publications page. I also changed the links so they open in a new, smaller window (like a dialog.)

Papers related to communication/networking middleware: IEEE Network

2008-10-15T13:41:00.002-04:00

Am on the trail of papers related to work on communications or networking middleware from a high-level perspective. Been reading IEEE Network publication online all day, and found some interesting issues as well as articles that may be quite relevant:

(2008) Vol 22, Issue 4: Context aware networks, in mobile scenarios (whole issue)
(2007) Vol 21, Issue 4: GE Network Systems Architecture (whole issue)
(2006) Vol 20, Issue 4: VON: a scalable peer-to-peer network for virtual environments (this paper has analysis of other P2P-based NVE systems)
(2006) Vol 20, Issue 3: A survey of transport protocols for wireless sensor networks
(2004) Vol 18, Issue 1: GE Middleware technologies for future communication networks (whole issue, and is very good)

Among other things:

Registered and uploading final version of paper for CCNC 2009 conference.
Testing the new WORKIT box, which may be the final version of the box we will ship

Grid computing and P2P resource discovery

2008-10-07T15:44:00.003-04:00

Excellent survey paper on various programming models in grid computing, which relates very closely to what I am looking into in finding programming models for the get/set and distributed network computing environment at large: "Grid Programming Models: Current Tools, Issues and Directions".

Another related work paper is "Peer-to-Peer Resource Discovery in Mobile Grids," which defines a P2P service discovery protocol/framework for mobile environments.

One other direction I am currently looking into is the relationship between grid computing, cloud computing and the mobile distributed computing I am working on...

Network programming: scalable web servers and Java NIO

2008-10-06T16:33:00.002-04:00

While looking for network programming APIs for my candidacy, I came across two excellent papers/presentations which are not directly related to my topic, but I need to bookmark in any case...

The first one is Scalable Network Programming: Or: The Quest For A Good Web Server (That Survives Slashdot) by Felix von Leitner. It talks about the problems of building scalable web servers, from ground up, including looking at fork() code, the time involved, measurements, comparison with threads, etc. An excellent read for people who know scaling web applications is hard, but would like to know more of why from an "under the hood" approach.

The second one, Improving Java Network Programming, by Brian Runk, is a simple application-level presentation, but compares the java.net and java.nio packages and discusses practical issues that arise while programming large-scale enterprise systems like ones built at Morgan Stanley.

Sockets over HTTP: Comet and Orbited

2008-10-01T11:47:00.002-04:00

After my talk on "Disruption Tolerant Applications in Opportunistic Networks" at the IRT meeting yesterday, I found some interesting info about sockets over HTTP using Javascript.

The Javascript implementation, called Orbited, claims to provide a TCP socket over a web browser. Sounds like a really interesting concept. A slightly outdated tutorial on writing an IRC client using Orbited.

This is related to a neologism called Comet, which is "a web application model in which a long-held HTTP request allows a web server to push data to a browser, without the browser explicitly requesting it." (quote from Wikipedia's entry on Comet.)

Related work: Market Contact Protocol

2008-09-26T10:47:00.002-04:00

An interesting Master's thesis called "Content Sharing for Mobile Devices" has an extremely comprehensive and complete look at the problem of finding and sharing content with other local mobile devices.

The thesis has excellent documentation comparing DHTs, overlay networks and other related work in this field.

The simulation and datasets in the thesis are also very comprehensive, and may be helpful if we ever do a simulation ourselves in this field.

The implementation itself, a protocol called "Market Contact Protocol" - at first glance - seems very similar to what we are doing with the BonAHA project, and related work such as LightPeers.

Nokia n800, Python, Bonjour and mDNS

2008-09-25T10:39:00.002-04:00

A project student has the task of developing useful 7DS-like applications on our lab's Nokia n800, so we went searching for how to develop applications on this platform.

It appears that Python is the Nokia n800's programming language of choice - tough luck for Java. So we would have to start working with Python and developing our framework in that language.

In order to use Bonjour on Python for the Nokia, we need the pyBonjour library.

We also need to install the Bonjour mDNSd daemon for handling mDNS packets - or Avahi, which is more supported on Linux. A look around reveals that Avahi is available on the Nokia n800 through the Canola project.

But for pyBonjour to work with Avahi, the binary-compatibility layer with Bonjour need to be installed [libdnssd1 and libdnssd-dev].

There are existing demo applications (one and two) on Nokia using Bonjour. Both seem to relate to web servers.

Some more info about Python and mDNS from razorvine.

Data-object vs communication P2P applications

2008-09-18T14:34:00.004-04:00

It appears that there are two sets of applications that BonAHA (or any ad-hoc network framework) will have to cover and address:

Data-object and file-oriented: In this sort of application, the most important topic are the files and the data themselves, not necessarily the location of these. E.g.: file sharing, file synchronization, chat
Person-to-person communication: In this sort of app, the focus is on the person to communicate with (not even the node itself, which is an abstraction, but the person). For example: drag-and-drop file sharing, instant message

So, BonAHA will have to address which of these two models it will be targetting - and if it can, and will, target both classes of applications.

DHTs actually handle the data-object model very well, expect that their setting up of an overlay network and additional network traffic in networks with heavy churn leaves much to be desired for MANET applications.

mDNS and DHT-related work

2008-09-18T14:34:00.002-04:00

This was a very productive week in looking forward to working more on the BonAHA framework and our architecture for disconnected ad-hoc networks.

mDNS and DNS-SD: Reading the specifications again, particularly focusing on their handling of:

Packet size: How does it deal with large packets? What is the maximum packet size?
iChat TXT record data: iChat exchanges its avatars using DNS-SD. But how large is it? What are the tradeoffs?

In terms of related work, here are some items I looked at this week that seem related to 7DS and the BonAHA framework:

P2PSIP RELOAD
System on Mobile Devices, SyD (A paper from this got accepted at Usenix 2004)
LIMO: But this doesn't actually seem to have much relationship to 7DS/BonAHA work - it seems to be an overall, comprehensive

Updated website + Fall courses

2008-09-02T12:37:00.003-04:00

As Fall semester rolls around, I decided to start it off by updating my website to use a new template - Concrete on FreeCSSTemplates.org. (PS: FreeCSSTemplates.org has some great, simple and totally free CSS templates for your website.)

I will end up taking "Analysis of Algorithms, I" this semester and may end up auditing the "Artificial Intelligence" course if possible...

WORKIT website now detects duplicate usernames; gives error messages

2008-08-22T13:41:00.003-04:00

The WORKIT website now gives error messages properly for two common reasons:

Duplicate username
Empty fields

I have put up the code here since Blogger's code highlighting system isn't too good.

Change to a new DB server

2008-08-19T15:30:00.004-04:00

Tis the day the WORKIT website moved to a new DB server, the shared MySQL server ... so we don't intrude on department MySQL server space. Requires a bit of recoding on my part to get the WORKIT website working with the new server.
Also, found that users can create duplicate entries with same username - that's a no-no. Need to fix this.
The Physics Machine Shop and I played phone tag today - we finally decided to meet on Thursday to finalize the hardware setup for the remaining baseline kits.

vi / vim extra tabs during paste - solved

2008-08-15T13:37:00.002-04:00

I've had this annoying problem with vi / vim ever since I started to use it - whenever you copy text and paste it into vi or vim, it indents each additional line with an extra tab, rendering it completely unreadable and requiring manual editing.

Luckily, I was able to find help on Google groups and the Wikia Vim pages. Solving this problem is as easy as typing in the command:

:set paste

That worked for me!

Wireless cards, antennas and connectors

2008-08-15T12:47:00.001-04:00

Today I hunted down a wireless antenna-cable combo to go with our wireless card, the Atheros-based Gigabyte GN-WIAG02.

I didn't realize that there were different forms of cables and antenna for wireless cards, so I had to learn more about these.

I found out on Hyper Link Tech that there are several types of these. Because the Gigabyte card is a 802.11 b/g, which runs on the 2.4GHz frequency, what I needed was a 2.4 GHz rubber duck antenna. 5 dBi seems to be fine for the WORKIT testbed.

There were some helpful pages on gumph.org and Seattle Wireless that helped me find out more about the antenna, cables and connectors that are needed for wireless.

It appears that the most common connector for the miniPCI based wireless cards is the MMCX connector.

Finally decided to get the AOpen wireless antenna, which appears to have the antenna as well as the cables needed to fit the antenna onto the kit.

Rich UIs in Java: themes and fade animations

2008-08-08T15:56:00.003-04:00

I was looking for ways to improve the look and feel of Java applications - for presenting my Java-based research work as a nice-looking app and for demos. I was inspired by some of the Adobe AIR themes out there, and I found some interesting stuff.

Among the many Swing Pluggable Look-and-Feel solutions is one that really stands out - Substance. It looks absolutely phenomenal, with a variety of options and customizations, plus its open-source. It easily stands out as a must-use PLAF for Swing. Take a look at some Substance screenshots.

There's that for themes ... but how about fade effects and transitions?

A little bit more searching reveals the animation layer, which includes transitions - in fact, automated transitions with no changes to app code! You need to see the above link for yourself to believe it.

Apparently, this fade animations is part of the laf-widget project now.

insmod, kernel versions, modules and Linux distros

2008-08-07T12:23:00.003-04:00

The joy of working with kernel versions and kernel modules.

So after spending a long time finally getting a modpost program running so I could build kernel modules, I was finally able to get them to compile. However, I found that the modules were not loading - insmod and modprobe gave a very unhelpful "-1 Invalid module format" error.

A little bit of searching around for "kernel modules" lead me to the helpful Linux Loadable Kernel Module HOWTO. A page titled Technical details provides the full details about how to find information about the module.

I was able to confirm that the module had compiled properly, and that it was atleast displaying the right meta-information. So what had gone wrong?

A search for the actual error message "invalid module format" led me to a forum discussion on Linuxquestions.org about insmod, where one poster suggesting using 'dmesg' to find out the details of the error.

Excellent information. A line in dmesg shows:

No module found in object
kaodv: version magic '2.6.20.1 preempt mod_unload 486'
should be '2.6.20-486-voyage preempt mod_unload 486'

Perhaps because I used the Debian Voyage kernel instead of the Voyage kernel sources? :( And after all the trouble I went through in finding a corresponding kernel source and building the tree and the toolkit...

Maybe there's a way to just rename just the kernel name, since everything else seems to match ... stay tuned.

The insmod / modprobe hurdle

2008-08-06T16:00:00.002-04:00

Today, I hit a brick wall with insmod/modprobe in installing/setting up AODV on the Linux kernel.

I was able to successfully compile the module, including the kernel object files. It appeared that the modpost compilation/linking worked perfectly as well.

However, a problem came up when running the start.sh script provided - it said that it could not find the "kaodv" module, which had been compiled in the current directory.

Perhaps it had not been loaded into memory properly. However, insmod /modprobe failed to find and load the module as well.

modpost can now compile kernel modules

2008-07-16T13:36:00.001-04:00

After my last post (which took quite a while to build all modules), modpost finally works and is able to compile kernel modules! Screenshot attached...

Long "voyage": modpost to work and compile

2008-07-16T11:20:00.003-04:00

Ah... after weeks of hacking through kernel code and Makefiles, I am finally able to compile and run modpost , for compiling my kernel modules on Voyage Linux 0.4.

This is the story: I need to install some additional kernel modules for the WORKIT project on Voyage. All of the kernel modules pass step 1 of the Makefile, and they compile, but then I get the error:

Building modules, stage 2.
MODPOST 1 modules
/bin/sh: scripts/mod/modpost: No such file or directory

A long search for modpost lead me to find that modpost is used for compiling modules, and is part of the kernel development tools (sorry, missing the link for that post, and couldn't find it again on Google - tells you how exotic this modpost is.)

After a lot more trial and error - which included downloading and trying to recompile the Voyage Linux kernels - I found two posts today that helped immensely in coming to the last step:

No modpost directory - Linux Forums : helped me realize how to build the modpost (but it was referencing the wrong directory; on my Voyage Linux, the code is at /lib/modules/2.6.20-486-voyage/build)

But I got the exact same error message as in the forum post: unrecognized command line option "-m".

I copied the makefile command, removed the -m directive, and tried compiling again and got a "missing elfconfig.h" error message and other errors dependent on that.

A search for "elfconfig.h" led me to a post of the Linux kernel mailing list about missing elfconfig.h which hints at running "make modules" and "make scripts".

Running those two commands on the /lib/modules/2.6.20-486-voyage/build directory solves the problem - it creates the elfconfig.h (!) file and compiles modpost. (Expect this to take a while - it makes all modules and scripts.)

CCNC BonAHA paper + iptables save

2008-07-11T13:14:00.003-04:00

Sorry, been missing a week since I was finishing up my BonAHA paper for CCNC 2009 ... along with finalizing the BonAHA SourceForge website to be ready for the paper submission.

I also discovered that for some reason, my iptables configuration outlined in And Linux is now a wireless AP router doesn't have each time I shutdown or reboot ... which means that I have to reenter it each time.

The usual way I find this out is by trying to connect a laptop to the AP to test it ... only to find that I am not able to access the Internet through it (I get an IP address fine, meaning DHCP works.)

And running "iptables-save" doesn't work either ... so apparently, I have to find another way to save the iptables configuration for the box.

Need ncurses, module-assistant to build Linux kernel?

2008-06-25T14:24:00.003-04:00

I think I may be getting somewhere with my compile problems for building Linux kernel modules for Voyage.

First of all, modpost apparently comes with linux-headers and is built when the kernel source is run with a "make" command.

Secondly, building the kernel apparently requires ncurses, which was causing most of my module build scripts to fail.

Several posts are recommending moving from modpost to module-assistant for Debian (and here) ... but alas, most modules I am working with are already configured to work with modpost. :( But something to keep in mind if I am writing my own kernel modules.

UPDATE, June 25, 2008 3:45 pm: I installed ncurses using this command:

apt-get install libncurses-dev

and I could indeed build the kernel without any problems.

And Linux is now a wireless AP router

2008-06-18T13:15:00.004-04:00

I followed the instructions in the YoLinux article on making the Linux gateway ... and it works perfectly. The WORKIT AP now functions as a gateway router, instead of the bridge as it was earlier. (The LinuxJournal article mentioned in my previous post is a little old at this point.)

I just had to follow the instructions to set up the gateway. This is what I used for the WORKIT AP (eth0 is my wired WAN connection and ath0 is the wireless LAN):

   # Delete and flush. Default table is "filter". Others like "nat" must be explicitly stated.
  # Flush all the rules in filter and nat tables
  iptables --flush           
  iptables --table nat --flush
  # Delete all chains that are not in default filter and nat table
  iptables --delete-chain    
  iptables --table nat --delete-chain

  # Set up IP FORWARDing and Masquerading
  iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
  iptables --append FORWARD --in-interface ath0 -j ACCEPT

  # Enables packet forwarding by kernel
  echo 1 > /proc/sys/net/ipv4/ip_forward

Also, while instructions exist for both iptables and ipchains, you will find yourself using the iptables rules since ipchains was RIP after Linux 2.4 came out.

Setting up a Linux AP: Bridge vs. router

2008-06-17T14:18:00.004-04:00

Now that I have DHCP and DNS set up, the last thing I need to do is set up the WORKIT box to act as a complete AP by bridging the wireless (LAN) and wired (WAN) connections.

I naively followed linux.com's instructions on setting up a wireless AP by bridging the connections.

I was surprised to see that it completely broke the DHCP/DNS/gateway system I had set up - my wireless client instead picked up the IP address (DNS and DHCP) from the department's regular network. And my "gateway" (the WORKIT AP) had "disappeared", since I was not being assigned an IP address from it and was not on its internal network (10.0.0.x) that I had set up.

What had happened? I finally realized I had set up Linux on the WORKIT AP as a glorified wireless bridge (think switch/hub), not a router!

What I now need to do is set up Linux to act as a gateway, so the WORKIT AP acts completely as a full AP.

A little more search leads to YoLinux's article on setting up a Linux gateway and on Linux Journal as well. Basically, this involves a little fiddling around with iptables...

Installing Kernel AODV for Linux

2008-06-17T11:38:00.002-04:00

I attempted to install the "Kernel AODV" kernel module today for the WORKIT AP. Progress was relatively smooth till the last step of the make process, which called "modpost" to create the kernel module.

As expected, there were some issues with my header include files, so had to change the Makefile and include the following lines:

CFLAGS += -I/usr/include/
CFLAGS += -I/usr/include/linux/
CFLAGS += -I/home/linux-2.6.20.1/include

(the last one is the custom directory where I untarred the kernel source.)

In addition, I also had to copy the following files from the kernel source directory:

irq_regs.h
device.h
pda.h

to /usr/include/asm, since the compiler was complaining that they were missing.

After moving all the files, the compilation went smoothly and produced all the .o files; however, it is now stuck at the last step of actually making it a kernel module using "modpost". More information coming soon.

Installing kernel source in Voyage Linux

2008-06-17T10:44:00.003-04:00

While making kernel based modules, I got an error of the type:

"Missing .config file in /lib/modules/2.6.20-486-voyage/build"

The reason this happens is because the kernel source is not installed with Voyage Linux, but if you are building kernel modules, you need access to this.

apt-get install kernel-sources does not work either.

What I did was:

Download the corresponding kernel source version from the Linux kernel archives.
Untar to /lib/modules/2.6.20-486-voyage/build
Run "make config"
Make sure to turn on "kernel support for .config files" (turned off by default, this is one of the first few options)

Now, you have a fully functioning kernel source tree in your Voyage Linux installation so you can compile kernel modules.

UPDATE, June 25, 2008: From a forum post, I found that you can actually find the source code for the Voyage Linux 0.4 kernel from their website at : http://www.voyage.hk/dists/0.4/linux/.

Actually, it looks like most of their distros have source online as well. Just replace "0.4" in the above link with your distribution number and expect to find your source tarball there.

Resolving compile problems with PSU's MAR - include directories and postgresql-dev

2008-06-13T14:21:00.003-04:00

Today I attempted to install PSU's Multiple Access Router component for WORKIT. However, I found that I had to set up the include directories in the makefile in order to get the component to compile.

These are the lines I added to the CONFIGURE that is used by the Makefile to get it to work:

CFLAGS += -I/usr/lib/gcc/i486-linux-gnu/4.1.2/include/
CFLAGS += -I/usr/include/c++/4.1.2/
CFLAGS += -I/usr/include/c++/4.1.2/i486-linux-gnu/
CFLAGS += -I/usr/include/linux/

In addition, I also had to install the Postgresql-dev package since it complained of "libpq-fe.h" missing. I installed it using:

apt-get install postgresql-dev

I did have to change one setting that was causing a conflict with the INCLUDE directories for the root Makefile. This file was /dotconf/Makefile, and I commented the following line:

# CFLAGS =

dnsmasq now working as DHCP server as well

2008-06-13T12:44:00.002-04:00

Now I have dnsmasq working as a DHCP server as well.

In my configuration, I added the following lines:

dhcp-range=ath0,10.0.0.10,10.0.0.250,24h
dhcp-leasefile=/var/tmp/dnsmasq.leases
dhcp-authoritative
# Gateway
dhcp-option=3,10.0.0.1
# DNS
dhcp-option=6,10.0.0.1

and dnsmasq is now able to serve up DHCP addresses on the fly to clients.

dnsmasq now works for DNS requests - after initial "query refused"

2008-06-13T11:27:00.005-04:00

For a long time, I was having a problem where dnsmasq refused to respond to DNS requests and gave an error message "query refused".

At first I thought that it might have been because dnsmasq wasn't running or not binding to the proper port, but after using "netstat -ap" and running "dnsmasq -d", it was obvious that dnsmasq was running and able to receive the queries.

So why wasn't dnsmasq failing to do IP lookups and interact with the upstream DNS servers? Even though from most documentation (such as on Debian and ubuntu.wordpress.com) suggests that a simple apt-get would do the trick?

Anyway, I found that my /etc/resolv.conf had only 127.0.0.1 listed as its nameserver, and dnsmasq was using this as its resolv.conf . I looked around to see if I could find another file that the resolvconf binary was generating that had the actual nameservers, and I found this file: /etc/resolvconf/run/interface/eth0

I change a line in /etc/dnsmasq.conf to read:

resolv-file=/etc/resolvconf/run/interface/eth0

And hey presto, it worked!

Network (eth0) automatically gets IP address using DHCP now

2008-06-12T13:13:00.003-04:00

I recently ran into a strange problem with the Voyage/Soekris box, where the box would not get the IP address using DHCP on bootup, even with the network cable plugged in. However, when I manually reset the networking (using ifdown/ifup eth0 or /etc/init.d/networking restart), it would work properly.

This was strange, as it had been working fine all along, and I had the /etc/network/interfaces set up as follows:

auto eth0
iface eth0 inet dhcp

After prolonged searching on the Internet, I found two entries on Ubuntu forums:

that helped, but didn't quite work in my case.

I finally added a line to the /etc/network/interfaces file to make it read like this:

auto eth0
iface eth0 inet dhcp
pre-up ifconfig eth0 up

and then, hey presto, the network card now picks up an IP address automatically on boot.

Fixing dnsmasq problems on Soekris/Voyage Linux

2008-06-12T10:21:00.003-04:00

(Sorry, my research blogging has been intermittent - but it does affect my documentation, so I am trying to be consistent about it.)

This morning, I needed to look at the DNS and DHCP servers on the Soekris box - somehow, it wasn't working perfectly.

The problem started when I first tried to install dhcpd (the DHCP server) last year without realizing dnsmasq had already been installed - and without knowing what dnsmasq is. After some frustrating error messages and doing some Google searches for dnsmasq and dhcpd, I finally found out more about how they work.

This article gives a good overview of how to set up dnsmasq.

Updates: LightPeers, WORKIT, Spring semester

2008-04-29T13:32:00.005-04:00

A few updates since my last blog post, and this is a quick summary:

I finally discovered a related work to BonAHA called LightPeers. LightPeers appears to be a PhD dissertation done in Denmark, and the aim of the project is the same: develop a robust application framework for mobile P2P or ad-hoc applications. There are a few differences between BonAHA and LightPeers, but the basic concept is the same.
I got the WORKIT MIPClient finally working and able to connect to a wireless AP! Yay! Now to get it connected to the WORKIT AP and test its connectivity to that system.
Almost at the end of the Spring semester now. That's good and bad ... because I'll be tied up for the next two weeks with project demos, TA grading and reports ...

Photos of Soekris net5501 in Demarctech ODE-8585 Enclosure

2008-04-18T14:37:00.004-04:00

At long last, I have photos of the Soekris net5501 in a Demarctech ODE-8585 enclosure.

This Soekris board is currently running Voyage Linux.

Thanks go to George at Columbia Univ's Physics Machine shop for putting this together.

Ad hoc applications

2008-03-31T13:27:00.004-04:00

For our 7DS project, we are dealing with a class of applications that are highly mobile, work in disconnected or intermittently connected networks, and without central servers such as DNS and DHCP.

What name do we give this new set of applications? We call them ad-hoc applications!

The only reference I was able to find for ad-hoc applications seems to suggest that this term isn't very popular, but maybe it will catch on...

Our 7DS system - and the BonAHA framework I am building at Columbia and has been put on SourceForge - will hopefully create a usable framework for these class of ad-hoc applications. We're nearly done with version 1.0, but I still need to fix up some bugs and document the code.

Drill Bushings from McMaster-Carr for WORKIT Soekris

2008-03-27T13:44:00.002-04:00

Hi, I'm back after a long while ... decided to continue using my Blogspot account for work related blogs, while the CS@CU wiki can remain as a wiki for official documentation.

Today, I was looking at finding ways of getting washers/bushings to seal off the Demarctech hardware that encloses our Soekris net5501 box. Talking to George at the machine shop indicates that McMaster-Carr would be the best source of such bushings.

I think I am looking for the items under :
Machining and Clamping > Drill Jig Bushings > Drill Bushings and Liners > Drill Bushings for Soft Materials

More to come soon ... as soon as I figure out exactly what I need.

Odd "Array" output in Joomla/PHP module solved

2006-08-24T16:12:00.000-04:00

I had an odd problem in which a Joomla! module I was writing was outputting "Array" after its output.

I didn't find any code in the module itself that was capable of outputting this error message, since this came after the last line of the module's output echo() statements. So, I was puzzled!

Then I realized that it might have been because I had overwritten a global variable. I had liberally used variables like $id, $content, $categories, etc. So I replaced all my variable names with ones that couldn't possible belong to the reserved variable list ... such as $category__id, etc.

And that solved the problem! :)

C libraries to capture layer two packets

2006-08-04T16:31:00.000-04:00

For the 7DS system, we need some libraries for peeking at the link layer packets. These are the ones we found:

Libradiate: http://www.packetfactory.net/projects/libradiate/
libpcap: http://www.tcpdump.org/
libnet: http://www.packetfactory.net/libnet/

Automake/autoconf for 7DS

2006-07-31T11:19:00.000-04:00

I finally started workng with the GNU Automake/Autoconf system for 7DS today. It's really neat! Far easier to work with than the faulty manual Makefiles I've been creating so far. I hope to have a fully GNU-style 7DS system (at least for my part) by the end of this week.

I found that the deadline for submitting my paper to INFOCOM has passed, so I will probably be submitting a paper to ICC07.

Verbose options and logging levels/priority

2006-07-26T12:56:00.000-04:00

Today I did something really cool with the 7DS logging system that enabled me to turn the "verbose" option off and instead introduce something more advanced: a level system that outputs log messages to the log file or screen based on the user setting.

To set this up, I basically made added two functions like this, one for on-screen and one for log files:


int SdsLogSetPrintLevel (int level)
{
    print_level = level;
}

and then changed the code so that it prints only the higher priority messages to log file or screen, depending on user settings.


  /* If level is greater than specified ,
   * then print to stderr. */
  if (level >= print_level)
    fprintf(stderr, logstatement);

  /* If level is greater than specified log level, write to logfile */
  if (level >= log_level)
    fprintf(logfp, "%s", logstatement);

Of course, I had to set the default values appropriately, so that the log file would get all messages.


int log_level = SDS_LOG_DEBUG,        /* File should have debug messages */
    print_level = SDS_LOG_INFO;        /* Screen should have info messages */

7DS code cleaning

2006-07-25T16:39:00.000-04:00

I' m continuing to clean up my 7DS code. I have to follow my advisor's instructions on formatting code, which are very helpful.

I'm also finding that other than configuration files and logging, I will well have to start using getopt() for command line options. Well, better late than never...

7DS software engineering and wikis

2006-07-18T15:58:00.000-04:00

Sorry I haven't been active for a while on this blog ... well, a few days ago, I realized I was running into a brick wall with the 7DS project, not sure of what to do, and faced with an overwhelming array of choices that I didn't know how to handle. :(

Luckily, I e-mailed Henning and met him today (with Se Gi and Andy) to get an idea of how to come to grips with the situation and proceed further with the project.

It looks like:

We will release a version that has the basic features, but working.
I think this can be released as source code with configure/make/make install, as well as installation binaries.
In parallel, we will draft guidelines for future versions and see how to make 7DS more modular so that it is "future-compatible". Things like the community extensions and file syncronization will be part of the next release.

In addition, Henning suggested that we use a wiki to document our work. That's a great idea! The only thing now is to find a place to host it ... unless we have a CS-wide one.

One great wiki I've heard a lot about is DokuWiki...

Installing Eclipse with PHP and C

2006-06-28T14:20:00.000-04:00

I decided to install Eclipse today to use as my primary development kit. I've used vi, gEdit, jExt (I really like jExt!) but life might be simpler with an IDE.

So I will try Eclipse with the C development and maybe PHP development environements and see how it works out.

Installed DarwinPorts, sqlite, swish-e, and all other required libraries

2006-06-27T17:09:00.000-04:00

OK - I finally managed to get DarwinPorts installed on AirTrain (the iMac). I couldn't get it installed under OS X 10.2 - it complained that TCL was built without threading enabled.

Now that I have OS X 10.4 and the Xcode tools, DarwinPorts installed without a hitch from source.

I am now able to successfully build all the other required libraries that I need to get 7DS running on the Mac, including:
- sqlite3
- swish-e
- libconfuse

(and maybe some more? I forget.)

Upgraded AirTrain iMac to Mac OS 10.4

2006-06-27T13:34:00.000-04:00

I upgraded AirTrain (the IRT lab iMac) from 10.2 to Mac OS X 10.4. Surprisingly, the install DVD had an upgrade option - so I didn't have to format the disk at all! So much for my warnings to my colleagues to backup all the stuff they needed...

Porting 7DS from Unix to Mac OS and Windows

2006-06-08T12:51:00.000-04:00

Right now, I'm involved in porting the 7DS system from the Linux system to run on Windows and the Mac OS X. It is really fun (!) though it might take a while to complete...

For Windows, I am currently trying Cygwin. I heard that it uses a compatibility layer and a cygwin.dll file though - meaning that Cygwin will have to be installed. I have heard about MinGW and will see if that avoids this issue.

For the Mac OS X, I first heard that we might need to install the XCode development tools, but now it looks like it is not necesssary. The Mac OS X 10.2 system I am working with already has the GCC environment, and except for dynamic library handling, it seems to compile most apps. Dynamic library loading is a problem though: I even installed dlcompat from Fink, but it still has problems! Fink, btw, is a build environment for Darwin that promises to make it easier for people to port open source apps to Mac OS.

Classification of XML data

2006-04-19T19:41:00.000-04:00

As part of the 7DS community extension, I will have to classify shared community XML objects using some sort of XML schema, RDF or RDF Schemas. We looked at OWL today, but that looks fairly complicated, so we may just go ahead and use RDF and RDF schemas.

PHP and web (http) servers on Windows Mobile / PocketPC

2006-04-14T14:34:00.000-04:00

As the scope of the 7DS project expands, I'm coming up against building more involved 7DS web applications. The search and multicast engines were C programs that produced binary CGI executables, but building more involved community-based web systems is going to be hard. :(

In looking to build the new applications in PHP, I was asked to see if PHP might be supported on Windows Mobile (one of our near-future development platforms) and I found it interesting that Windows Mobile SDK has its own HTTP/web server.

Even more interesting, there have been problems porting the Zend PHP engine to Windows Mobile, so there is an alpha version of a ground-up PHP engine built for Windows Mobile with its own web server.

Always use sizeof() when malloc()ing

2006-03-31T13:44:00.000-05:00

I've had some real big problems implementing my webpage retreiver, and I fixed it once I realized that I had to include the sizeof() everytime I malloc()d.

So remember to use this everytime you malloc, say a string:


newstring = malloc ((strlen(oldstring)+1) * sizeof(char));

Remember to add 1 as well, just as I did: C strings terminate with a '\0', which is an extra character.

Memory leak for null string assignment

2006-03-30T18:58:00.000-05:00

This is really wierd... the following piece of code in my 7DS system results in a huge memory leak, gobbling up memory really fast.


  if (0 == hits) {
    // No results, empty xmlResults and return
    sprintf (xmlResult, "");
    return 0;
  }

Disabling it solves the memory problem - I wonder why?

malloc() and free() error for dynamic strings solved

2006-03-28T16:00:00.000-05:00

OK, I am a newbie at dynamic strings in C, so please forgive my silliness.

I have been getting "*** glibc detected *** free(): invalid next size (fast)" errors in my application that has to create dynamic path names and couldn't figure it out.

Finally I did a Google search and found the solution here: http://www.eskimo.com/~scs/cclass/int/sx7.html

Guess what I had done? malloc()d the string to use one less character than needed like this:
escapedURL = malloc (strlen (URL));

This is CORRECT:
escapedURL = malloc (strlen (URL) + 1);

Because C strings end with a '\0' character.

Parsing filename and directories out of given path in C

2006-03-27T14:13:00.000-05:00

Sample code for how to parse directory structure and path names for a given path string in C. I will use this for the webpage retreiver project I am working on.


#include <string.h>
#include <stddef.h>
#include <libgen.h>
#include <stdio.h>

/* This program parses the path given in the argument into directories
* and filename, creates the directory structure and creates an
* empty file as well */
int
main (int argc, char **argv)
{
const char delimiters[] = "/\\"; /* File delimiters */
char *token, *oldtoken, *cp;
const char program_directory = getcwd (NULL, 0);  /* Program directory */

/* Create copy of path string */
cp = malloc (strlen (argv[1]));
strcpy (cp, argv[1]);

/* Split string into tokens */
token = strtok (cp, delimiters);

printf ("Directory = ");

/* While token is not NULL */
while (1)
{
oldtoken = malloc (strlen (token));
strcpy (oldtoken, token); /* Copy current token */
token = strtok (NULL, delimiters); /* Go to the next token */
/* If nexxt token is NULL, it is the last part and assumed to
* be a filename */
if (token == NULL)
 {
   printf ("\nFilename = %s\n", oldtoken);
   /* Create an empty file of that name */
   FILE *fp;
   fp = fopen (oldtoken, "w");
   fclose (fp);
   break;
 }
/* Otherwise it is a directory */
else
 {
   printf ("%s ", oldtoken);
   mkdir (oldtoken, 0755); /* Create the directory */
   chdir (oldtoken); /* Go there */
 }
 free (oldtoken);
}
printf ("\n");
/* Free any memory elements */
free (cp);
oldtoken = NULL;

chdir (program_directory);
}

URL or URI parsing in libxml and c

2006-03-17T14:25:00.000-05:00

I found out that libxml has URI functions that will allow you to parse URIs using libxml.

Here's an example program:


#include <stdio.h>
#include <libxml.h>

int main(int argc, char **argv) {

/* Create a null URI */
xmlURIPtr url = xmlCreateURI ();

/* Parse the user input URI */
url = xmlParseURI ( (argc <= 1) ? "http://www.theepochtimes.com/" : argv[1]);

/* Print all the respective information */
printf ("scheme = %s\n", url->scheme);
printf ("opaque = %s\n", url->opaque);
printf ("authority = %s\n", url->authority);
printf ("server = %s\n", url->server);
printf ("user = %s\n", url->user);
printf ("port = %d\n", url->port);
printf ("path = %s\n", url->path);
printf ("query = %s\n", url->query);
printf ("fragment = %s\n", url->fragment);
printf ("cleanup = %d\n", url->cleanup);
}

Parsing HTML using tidy and tidylib

2006-03-17T13:22:00.000-05:00

It's so hard to find a C program on the web that can parse HTML! Yes, you can find parsers written in Perl and other languages, but not C!

So I might as well share what I've learnt so far. I am making the 7DS HTML parser in libxml, but I experimented using tidy and tidylib as well, and here's how the code for that looks:


#include <tidy.h&rt;
#include <buffio.h&rt;
#include <stdio.h&rt;
#include <errno.h&rt;

/**
* Dump the list of nodes and their attributes
* Modified from tidylib documentation
*/
void dumpNode( TidyNode tnod, int indent )
{
TidyNode child;

for ( child = tidyGetChild(tnod); child; child = tidyGetNext(child) )
{
ctmbstr name = tidyNodeGetName( child );
if ( !name )
{
  switch ( tidyNodeGetType(child) )
  {
  case TidyNode_Root:       name = "Root";                    break;
  case TidyNode_DocType:    name = "DOCTYPE";                 break;
  case TidyNode_Comment:    name = "Comment";                 break;
  case TidyNode_ProcIns:    name = "Processing Instruction";  break;
  case TidyNode_Text:       name = "Text";                    break;
  case TidyNode_CDATA:      name = "CDATA";                   break;
  case TidyNode_Section:    name = "XML Section";             break;
  case TidyNode_Asp:        name = "ASP";                     break;
  case TidyNode_Jste:       name = "JSTE";                    break;
  case TidyNode_Php:        name = "PHP";                     break;
  case TidyNode_XmlDecl:    name = "XML Declaration";         break;

  case TidyNode_Start:
  case TidyNode_End:
  case TidyNode_StartEnd:
  default:
    assert( name != NULL ); // Shouldn't get here
    break;
  }
}
assert( name != NULL );
char whitespace[indent];
memset (whitespace, ' ', indent);
whitespace[indent-1] = '\0';
//    printf( "%sNode: %s\n", whitespace, name );

/* Get the first attribute for all nodes */
TidyAttr tattr = tidyAttrFirst (child);
while (tattr != NULL) {
  /* Print the node and its attribute */
  printf ("%s  %s %s= %s\n", whitespace, tidyNodeGetName (child), tidyAttrName (tattr), tidyAttrValue (tattr));
  /* Get the next attribute */
  tattr = tidyAttrNext (tattr);
}
dumpNode( child, indent + 4 );
}
}

/* Dump the whole document */
void dumpDoc( TidyDoc tdoc )
{
dumpNode( tidyGetRoot(tdoc), 0 );
}

/* Dump only the body */
void dumpBody( TidyDoc tdoc )
{
dumpNode( tidyGetBody(tdoc), 0 );
}

int main(int argc, char **argv )
{
/* Input file: Either the first argument or "../test.html" */
const char* input = (argc > 1) ? argv[1] : "../test.html";
TidyBuffer output = {0};
TidyBuffer errbuf = {0};
int rc = -1;
Bool ok;

TidyDoc tdoc = tidyCreate();                     // Initialize "document"
printf( "Tidying:\t%s\n", input );

ok = tidyOptSetBool( tdoc, TidyXhtmlOut, yes );  // Convert to XHTML
if ( ok )
rc = tidySetErrorBuffer( tdoc, &errbuf );      // Capture diagnostics
if ( rc >= 0 )
/* Read from the HTML file */
rc = tidyParseFile( tdoc, input );           // Parse the input
if ( rc >= 0 )
rc = tidyCleanAndRepair( tdoc );               // Tidy it up!
if ( rc >= 0 )
rc = tidyRunDiagnostics( tdoc );               // Kvetch
if ( rc > 1 )                                    // If error, force output.
rc = ( tidyOptSetBool(tdoc, TidyForceOutput, yes) ? rc : -1 );
if ( rc >= 0 )
rc = tidySaveBuffer( tdoc, &output );          // Pretty Print

if ( rc >= 0 )
{
if ( rc > 0 )
  printf( "\nDiagnostics:\n\n%s", errbuf.bp );
printf( "\nAnd here is the result:\n\n%s", output.bp );
}
else
printf( "A severe error (%d) occurred.\\n", rc );

tidyBufFree( &output );
tidyBufFree( &errbuf );

/* Now parse and print the tags in the HTML document */
dumpDoc (tdoc);

tidyRelease( tdoc );
return rc;
}

Webcrawler using libxml, libcurl and tidy

2006-03-14T12:55:00.000-05:00

Contrary to my writeup in the last post about how wget might be the best way to webcrawl and fetch files to a local cache, my thoughts now are different.

You can use the following libraries to build a decent webcrawler:

1. Tidy: Use tidylib to clean up your HTML pages and make them XHTML. tidylib's webpage has sample code that is good enough for converting HTML to XHTML - just make sure you save to a file using tidySaveFile().

libxml has problems parsing HTML, even if used with xmlRecoverFile() rather than xmlParseFile().

2. libxml: Parse the XHTML, get all elements' attributes (and any other URLs you need) and pass on the URLs to libcurl to download. Need I say more?

Well, actually I should. libxml is a little hard to understand from the API, and sample code to do what you want is hard to find. I had to do quite a bit of searching, looking up sample programs, and then reading the API to figure out how things worked.

3. curl: Or rather libcurl. To retrieve files from the Net. Again, need I say more?

Life would have been simpler if curl had a recursive download function ... or wget had a library I could use ... but then, that's why we computer engineers and students have a life!

wget to create local cache of webpage

2006-02-28T13:10:00.000-05:00

Here's how to use wget to create a local cache of a webpage:
wget -r -l 1 –p –-convert-links http://www.yourdomain.com

For more information: http://www.devarticles.com/c/a/Web-Services/Website-Mirroring-With-wget/1/

2006-02-14T12:47:00.000-05:00

Very nice and detailed documentation on creating RPMs:
http://www.gurulabs.com/GURULABS-RPM-LAB/GURULABS-RPM-GUIDE-v1.0.PDF

It includes information not present in the official RPM FAQ at http://www.rpm.org/

Processes in C: /dev/urandom C code, forking several child processes...

2006-02-06T13:01:00.000-05:00

Hi,

Sorry, I've been busy with courses AND research lately, it 's given me less time to post on the blog.

For an OS course, we have to do the Miller-Rabin test for primality, and it involves forking atleast 3 child processes; getting random numbers from /dev/urandom, etc.

Here is some code for having 2 or more child processes:
http://www.csl.mtu.edu/cs4411.ck/www/NOTES/process/fork/fork-03.c

For reading from /dev/urandom (and why that's a good idea):
http://www.kalyanvarma.net/tech/security/random.html

Got Darwin Streaming Server running in Linux

2005-12-13T13:47:00.000-05:00

Buggered if I know why ... but I installed Darwin Streaming Server without a root account and changed the settings and other files as needed.

Then, yesterday, it worked when I started it, but then , I shut it down and couldn't start it again.

Then, after many tries, it finally worked when I chdir'd to /homes/streaming/usr/local/sbin and ran this command:
./DarwinStreamingServer -c /homes/streaming/etc/streaming/streamingserver.xml

But it wouldn't run from any other directory!

The web admin page still does not work; wonder why.

PhD qualifiers & Installing Darwin without root account

2005-12-12T14:10:00.000-05:00

Sorry, I've disappeared for a while since I'm preparing for the EE department's PhD qualifying exams.

I still have to install Darwin Streaming Server for the CS department's videos. My colleague Salman tipped me off to this great link on how to install Darwin without a root or su account.

But the new Darwin SS has some changes from the link that is posted. You might want to do the following instead:
- Edit the Install script so that the directories are your directories
- In Install script, comment the lines to create user qtss and the chmod lines
- Change the streamingserver.xml he mentions to update the directory links
- The new streamingadminserver.pl has default configuration values in the script itself, so you have to change those values rather than change an external config file.

If you are using vi, you can do the replacements by issuing the following commands:


:%s/\/usr\//\/homes\/streaming\/usr\//g
:%s/\/var\//\/homes\/streaming\/var\//g
:%s/\/etc\//\/homes\/streaming\/etc\//g

If you are editing streamingserver.pl, you ONLY need to do that for lines 230 - 270 in the current version, because only those are Linux-specific config settings. (Replace % with 230, 270)

I basically diff'd the config and script files he had with mine and decided to just play it safe and modify the default files.

Other than that, you should hear more from me after I'm done with my PhD quals in early January ... but at that point, I'll be taking Operating Systems and a seminar course for the Spring 2006 semester.

NetSurf - Source code for browser

2005-11-07T16:19:00.000-05:00

Right after I posted the last post about lack of open-source documentation for HTML parsing, I stumbled across this neat browser for RISC OS: NetSurf. And the full source code is neatly documented!

I plan to go over the code for NetSurf in building my new app.

libxml for parsing HTML

2005-11-07T15:47:00.000-05:00

Long time no see. Have been busy with preparing for my EE PhD qualifiers!

Well anyway, now I am to create a HTTP-retreiver-&-parser to get files for the 7DS multicast query system. It now has to not only get the result-set for the 7DS queries, but should also get the files themselves, as well as associated elements, such as images, etc.

I found several HTML parsers for C (after long searches) such as ekhtml (nil documentation), tidy (library does not build properly) and LibWWW (supposed to be very complicated) ... and have settled on using LibXML's inbuilt HTML parsing tools.

Sad that open source code has very little documentation ... hey, but neither does 7DS yet!

Reformatting source

2005-10-10T13:58:00.000-04:00

I came back from India 2 weeks back, and am spending time reformatting my C source code.

I am going to use a tool called DOC++ for documentation. Also, I am going to split my code in such a way that commonly used functions go into a "library" file.

Wireless card woes

2005-08-11T14:47:00.000-04:00

I'm still having problems getting the Senao wireless card to work on WRAP. :(

I am coming to the conclusion that maybe ... there is something wrong with the device itself?

I finally managed to get a copy of lspci for Bering and ran it on WRAP - I got it here http://fritzfam.com/brad/leaftmp/

But now that I am running lspci, I don't see any wireless card on the PCI list!! :(

My last hope - that it is a PCI card with a PCMCIA - PCI bridge ... but there's not very much hope.

7DS works on WRAP

2005-08-11T14:42:00.000-04:00

I got 7DS working on WRAP now!

I've mostly focussed on getting the wireless card to work with WRAP for the past few weeks... and in the meantime, I did some work on getting 7DS running on the WRAP board.

Here are what I did:
- Put all the 7DS binaries into 7ds.lrp
- Ran the query_receiver and it received multicast packets sent by another machine
(This was a little wierd; multicast did not work one day, but it worked the next.)

- Copied the directory structure using /home/local/sumans/ (same as on the development machines) and put this into 7ds_cach.lrp
- Copied the db.7ds.queries sqlite3 database to the appropriate place on the WRAP board

and it worked!

Also, I got the 7DS web interface and CGIs working as well. Here's how that was done:
- The website and CGIs are already in 7ds_cach.lrp
- I had to modify the mhttpd setting on WRAP and add this line:
dir = /home/sumans/local/www

and that worked as well!

natsemi network card on WRAP

2005-07-25T13:18:00.000-04:00

Well, I got the NatSemi network card working on the WRAP board after a lot of confusing tries!

First of all, I had to uncomment the lines for "crc32" and "natsemi" to load the modules at start up - which I forgot to do.

Once I did that, I got some "unresolved symbol" error messages. A post to the LEAF mailing list revealed that I might have different versions of the kernel and modules - which was the case! I had downloaded the latest version of kernel and modules, but used an older version of the WRAP-specific kernel ... which is where the problem was.

I then updated the WRAP board to use the same versions of the kernel and modules, and it worked!

Not only did the natsemi driver work, but I also got the 7DS multicast network working today! It didn't work last week (CRF told me the systems I was using were behind "different switches") but they sure do work now.

WRAP, wireless cards and LEAF

2005-07-14T15:18:00.000-04:00

Now that 7DS runs on LEAF/WRAP, I need to figure out how the wireless card on WRAP works.

First off, the boot up screen for Bering does not show that a wireless card has been detected.

PCI: Device 00:95 not found by BIOS

Bad news? :(

After taking a close look at the miniPCI board on the WRAP board, I made out the words "SL-2511MP PLUS 2.10". A Google search reveals that it is a Senao 802.11b wireless card.

Well, I should have known this because it on the purchase invoice, but I like to do things the hard way. :(

How do we get LEAF to recognize this?

A few more hours on Google leads me to dozens of discussions on mailing lists, madwifi, hostap and Jean Tourrilhes's Linux Wireless page - which don't help me very much, because Senao is not mentioned anywhere.

But I have gotten some idea now of how wireless works on Linux.

The Senao website itself is not helpful. :(

A few more hours on Google lead to this excellent introduction to the Senao wireless cards by SeattleWireless - I learn that the Senao card is built around the Prism 2 chipset. They also say the HostAP drivers work.

Well, that's something that should get me started on getting the wireless card working...

And neat - I just found some detailed documentation of setting up a wireless card based on the Prism wireless card at skippy.net!

Made glibc 2.3 lrp package

2005-07-14T15:12:00.000-04:00

I threw in the towel with uCLibc and created a glibc 2.3.2 LRP package. I also created LRP packages for libm and libpthread that are glibc based.

And hey - 7DS works on LEAF Bering-uCLibc/WRAP now! (Will try to add screenshots soon.)

The reason I decided not to use uCLibc was because:

I couldn't run it on the Linux box I was building under because I am not root - so no /lib/ld-uclibc.x.x (wonder why the links are hardcoded)
When I finally managed to transfer a uCLibc-compiled version of a simple program to the WRAP board, two things happened:

First, it complained that libgcc_x.so.1 was missing.
After I manually copied that library file, my program gave segmentation faults that I couldn't fix, because LEAF has no debugger.

So I went back and created my own glibc 2.3 LRP packages. I referred to the files that were in the glibc 2.2.5 package and created one - and it worked!

Compiling with uCLibc (and) CS MICE

2005-07-13T13:51:00.000-04:00

I downloaded ucLibc's toolchain (apparently, you have to build a whole new toolchain called "buildroot" to compile programs with uClibc, but that wasn't too hard.)

Then I built it and compiled a simple program ... and the compile worked. Yay!

But the program wouldn't run because it was looking for /lib/ld-uclibc.x.x - which I couldn't install because I am not root. :(

There was something in uCLibc documentation about the paths being hardcoded, wonder why.

---------

Also, met with CU-CS webmaster Ben today - talked to him about how to use the MICE system. I will be hacking some PHP scripts to see if I can read data from MICE.

Eventually, this should develop into a Mambo component that can be plugged into the new Mambo version of the IRT website.

7DS on WRAP: ucLibc

2005-07-11T15:13:00.000-04:00

My persistent efforts in making 7DS run on the LEAF/WRAP board hit a dead end last week.

The reason was that the latest glibc LRP package they have is 2.2.5 ... but my build requires 2.3!

I thought of creating an LRP package of glibc 2.3, but that may turn out to be a monster library!

My professor suggested last week that I reconsider and see if I can instead compile 7DS with ucLibc to get it running on WRAP, so I am taking a look at that...

Bering-ulibc installed on WRAP

2005-06-29T14:16:00.000-04:00

Phew, after hours upon hours, I finally succeed in installing Bering-uLibC on the WRAP board!!!

First off, I couldn't just install and run it as I did with WISP ... Bering-ulibc is a different animal altogether.

There's a ton of documentation at this page: LEAF for the pcengines WRAP but some of it might need to be tweaked for your needs. What I did was basically as follows (on Windows XP machine with CF card reader):

Popped the CF into memory card reader.
Formatted using WinXP DOS format (format d: /FS:FAT)
Note: CF needs to be formatted as FAT12 or FAT16, otherwise it won't boot. And don't worry about having to make a DOS 6.22 boot diskette; WinXP DOS's format works just fine - just make sure to specify /FS:FAT to format it as FAT16.
Download syslinux (http://syslinux.zytor.com/) and unzip the latest version on your drive.
Run syslinux -s d:
You should have already downloaded the latest Bering-ulibc package from the download website of the LEAF website.
Unzip the image and use WinImage to open the .imz file.
Note: I wasn't able to unzip their .imz files using any software other than WinImage.
Unzip the Bering files to a local directory.
(All following instructions use the same directory.)
Download any additional packages and put them in the same directory
Edit leaf.cfg in this directory and make following changes:
- Change /dev/fdxx to /dev/hda
  Note: The documentation says /dev/hda1, but I spent many hours trying to get that to work ... so when I changed it to /dev/hda, it finally worked. Perhaps WRAP uses /dev/hda because there is only one CF without partitions.
- Append the additional packages to the LRP list so that they also boot up.
Edit syslinux.cfg and make the following changes:
- Change /dev/fdxx to /dev/hda
Download the kernel with patch for WRAP that does not require a keyboard controller. Delete the old "linux" file and rename the newly downloaded "linux-2.4.x-xx.tux" to "linux".
Download the initrd-ide-cd file that will allow you to boot from the CF rather than a floppy. Delete the old "initrd" file and rename your "initrd-ide-cd" to "initrd".
Copy all your files to the CF.
Note: You will notice you have an ldlinux.xx on your local directory as well as the CF. Leave the one on the CF intact - it was created by syslinux and you need that to boot the CF.

Remove the CF (after unmounting it first if needed), pop it into the WRAP board, and watch WRAP boot up with Bering-ulibc and your required packages!

Edit [July 14, 2005]: One step I missed mentioning here is that you have to make the serial port the primary means of input/output for communicating with the device. Here's some information from the LEAF documentation on how to use a serial port with LEAF.

In short, you have to edit syslinux.cfg (use 9600 baud, I had some trouble at higher speeds) and /etc/inittab. The /etc/inittab is the toughie, as you have to unpack the etc.lrp package, edit the file, repackage the LRP and copy it back to your CF.

India trip; Mozilla search

2005-06-28T17:19:00.000-04:00

Today I have had to virtually cancel my India trip due to a lot of reasons. :( I don't expect to be going this year ... hopefully next year.

For the 7DS project, I have had to look at seeing how Mozilla's toolbar search works. It looks like it has to be built as a plugin for Mycroft. (Don't be scared by the word plugin - looks like simple "plugins" can be built using XML files.)

Another interesting thing is the keyword search from the address bar! Sounds interesting...

Other updates: WRAP is now going to use Bering-uclibc instead of WISP, since the Bering kernel is more recent than the WISP kernel, and also includes the wd1100.o driver by default, which should turn off the "keyboard jammed" messages.

I've had some trouble getting the Bering image to boot off my CF, but I think it's just a problem of not reading the documentation...

I'm sure there are other updates related to 7DS over the last week or so, but I've plain forgotten...

Getting the hang of LEAF

2005-06-07T17:31:00.000-04:00

Whew, I am finally beginning to get the hang of LEAF, how to install packages, etc (at least theoritically.)

A lesson: for best documentation on open source projects, refer to the mailing lists, not the official website or documentation!

An aside: I've been looking at BREW and J2ME for porting our 7DS project to cell phones ... but it appears that J2ME beats BREW hands down? More on this to come.

sqlite time and date, garbage collection

2005-06-03T11:23:00.000-04:00

Phew, I _finally_ managed to figure out how to garbage collect old values in a sqlite database.

There wasn't much documentation on the date/time functions in sqlite, so I had to do quite a bit of reading on that, strftime() as well as locale times (because that heavily influences the strftime () values.)

Anyway, here is the magic code I used:
DELETE FROM queries WHERE (strftime('%s', time_created) + keepalive*60 - strftime('%s','now', 'localtime') < 0)

where my table 'queries' has columns 'query', 'time_created' and 'keepalive' (in minutes).

The formula is obvious:
if (time_created + keepalive - current_time < 0) {
delete the query;
}

and that's what that SQL does.

Porting Linux to LEAF

2005-05-26T16:12:00.000-04:00

Did I ever tell you how hard it is to port Linux applications to embedded systems?

Well, apparently to port your application to LEAF (running on WRAP with x86 and all), you still need to repackage all our libraries, binaries, etc into a LRP format (a highly compressed .tar.gz file).

So far, so good, but what happens when you have a ton of libraries to port to the WRAP? Like libxml, swish-e ... tinyHTTP?

And also, the documentation for making LRP packages is not that good, so you mostly have to find your way through it...

More on this to come.

leaf-wisp and 7DS queries

2005-05-13T14:33:00.000-04:00

I removed m0n0wall from the WRAP box and "installed" a LEAF distro (branch actually) called "Wisp" which is a small version of LEAF meant for CF cards.

I just downloaded the binary and wrote it to the CF card, and it works perfectly - except for the silly and well-documented "pc_keyb: keyboard jammed" error message. Other than that, it seems to work smoothly - boots properly and gives me a shell prompt when I quit the configuration screen.

The 7DS query page is finished: see it in all its glory in HTML! (Page subject to change.)

Of course, it is far from perfect, and I still need to work more on tying the different pieces together and making them robust.

XML parsing and more on WRAP

2005-05-11T15:21:00.000-04:00

I finally managed to get the XML parsing with libxml working to my heart's content. For one, using the -lxml apparently uses the old libxml libraries, so I tried compiling my programs with -lxml2 and that solved a lot of compiler and linker problems.

When I finally got XML working, my GNU time functions stopped working for some odd reason. gdb told me that my program was crashing on the localtime() call - while it had been working perfectly all along! Argh!

As for WRAP box updates - I found out last week that m0n0wall, neat as it is, has no shell and a very limited interface which only allows you to change IP address and a few other elementary settings.

Our idea of installing 7DS on m0n0wall on WRAP goes out the window!

But its sibling m0n0box seems interesting and more fully featured, though I would like to see if we can use Linux on WRAP ...

libxml & sqlite

2005-05-04T17:13:00.000-04:00

Well, I finished sorting out sqlite and am able to work with it using C programs quite well. The callback functions are a little tricky, but I am getting the hang of them.

I also started using libxml today to encode and parse all those query and response messages. It's really neat and easy, but the code for parsing something is long and complicated! Despite that, working with XML seems to be fun...

More on sqlite and WRAP

2005-05-02T14:20:00.000-04:00

Well, I finally managed to get sqlite working with my C program.

Installation was easy, and I did that last week. The C API was a little harder, but I got it working today.

I got the female-female null modem adapter and am going to use it to play with the m0n0box / WRAP system today.

NSF progress report and SQLite

2005-04-27T17:16:00.000-04:00

Today I worked on (and finished) a NSF progress report for the components of the 7DS system that I worked on.

I also started playing with sqlite and reading through documentation for it. I need a database for my query-multicast system for 7DS, and it looks like this may be the right fit.

First post

2005-04-21T13:00:00.000-04:00

This blog will be about my Computer Science research in the IRT Lab at Columbia University. I am hoping it will help others who struggle with complex hardware and software stuff... and also serve as a reminder for myself when I need to write documentation.