Got Darwin Streaming Server running in Linux

Buggered if I know why ... but I installed Darwin Streaming Server without a root account and changed the settings and other files as needed.

Then, yesterday, it worked when I started it, but then , I shut it down and couldn't start it again.

Then, after many tries, it finally worked when I chdir'd to /homes/streaming/usr/local/sbin and ran this command:
./DarwinStreamingServer -c /homes/streaming/etc/streaming/streamingserver.xml

But it wouldn't run from any other directory!

The web admin page still does not work; wonder why.

PhD qualifiers & Installing Darwin without root account

Sorry, I've disappeared for a while since I'm preparing for the EE department's PhD qualifying exams.

I still have to install Darwin Streaming Server for the CS department's videos. My colleague Salman tipped me off to this great link on how to install Darwin without a root or su account.

But the new Darwin SS has some changes from the link that is posted. You might want to do the following instead:
- Edit the Install script so that the directories are your directories
- In Install script, comment the lines to create user qtss and the chmod lines
- Change the streamingserver.xml he mentions to update the directory links
- The new streamingadminserver.pl has default configuration values in the script itself, so you have to change those values rather than change an external config file.

If you are using vi, you can do the replacements by issuing the following commands:

:%s/\/usr\//\/homes\/streaming\/usr\//g
:%s/\/var\//\/homes\/streaming\/var\//g
:%s/\/etc\//\/homes\/streaming\/etc\//g

If you are editing streamingserver.pl, you ONLY need to do that for lines 230 - 270 in the current version, because only those are Linux-specific config settings. (Replace % with 230, 270)

I basically diff'd the config and script files he had with mine and decided to just play it safe and modify the default files.

Other than that, you should hear more from me after I'm done with my PhD quals in early January ... but at that point, I'll be taking Operating Systems and a seminar course for the Spring 2006 semester.

NetSurf - Source code for browser

Right after I posted the last post about lack of open-source documentation for HTML parsing, I stumbled across this neat browser for RISC OS: NetSurf. And the full source code is neatly documented!

I plan to go over the code for NetSurf in building my new app.

libxml for parsing HTML

Long time no see. Have been busy with preparing for my EE PhD qualifiers!

Well anyway, now I am to create a HTTP-retreiver-&-parser to get files for the 7DS multicast query system. It now has to not only get the result-set for the 7DS queries, but should also get the files themselves, as well as associated elements, such as images, etc.

I found several HTML parsers for C (after long searches) such as ekhtml (nil documentation), tidy (library does not build properly) and LibWWW (supposed to be very complicated) ... and have settled on using LibXML's inbuilt HTML parsing tools.

Sad that open source code has very little documentation ... hey, but neither does 7DS yet!

Reformatting source

I came back from India 2 weeks back, and am spending time reformatting my C source code.

I am going to use a tool called DOC++ for documentation. Also, I am going to split my code in such a way that commonly used functions go into a "library" file.

Wireless card woes

I'm still having problems getting the Senao wireless card to work on WRAP. :(

I am coming to the conclusion that maybe ... there is something wrong with the device itself?

I finally managed to get a copy of lspci for Bering and ran it on WRAP - I got it here http://fritzfam.com/brad/leaftmp/

But now that I am running lspci, I don't see any wireless card on the PCI list!! :(

My last hope - that it is a PCI card with a PCMCIA - PCI bridge ... but there's not very much hope.

7DS works on WRAP

I got 7DS working on WRAP now!

I've mostly focussed on getting the wireless card to work with WRAP for the past few weeks... and in the meantime, I did some work on getting 7DS running on the WRAP board.

Here are what I did:
- Put all the 7DS binaries into 7ds.lrp
- Ran the query_receiver and it received multicast packets sent by another machine
(This was a little wierd; multicast did not work one day, but it worked the next.)

- Copied the directory structure using /home/local/sumans/ (same as on the development machines) and put this into 7ds_cach.lrp
- Copied the db.7ds.queries sqlite3 database to the appropriate place on the WRAP board

and it worked!

Also, I got the 7DS web interface and CGIs working as well. Here's how that was done:
- The website and CGIs are already in 7ds_cach.lrp
- I had to modify the mhttpd setting on WRAP and add this line:
dir = /home/sumans/local/www

and that worked as well!

natsemi network card on WRAP

Well, I got the NatSemi network card working on the WRAP board after a lot of confusing tries!

First of all, I had to uncomment the lines for "crc32" and "natsemi" to load the modules at start up - which I forgot to do.

Once I did that, I got some "unresolved symbol" error messages. A post to the LEAF mailing list revealed that I might have different versions of the kernel and modules - which was the case! I had downloaded the latest version of kernel and modules, but used an older version of the WRAP-specific kernel ... which is where the problem was.

I then updated the WRAP board to use the same versions of the kernel and modules, and it worked!

Not only did the natsemi driver work, but I also got the 7DS multicast network working today! It didn't work last week (CRF told me the systems I was using were behind "different switches") but they sure do work now.

WRAP, wireless cards and LEAF

Now that 7DS runs on LEAF/WRAP, I need to figure out how the wireless card on WRAP works.

First off, the boot up screen for Bering does not show that a wireless card has been detected.

PCI: Device 00:95 not found by BIOS

Bad news? :(

After taking a close look at the miniPCI board on the WRAP board, I made out the words "SL-2511MP PLUS 2.10". A Google search reveals that it is a Senao 802.11b wireless card.

Well, I should have known this because it on the purchase invoice, but I like to do things the hard way. :(

How do we get LEAF to recognize this?

A few more hours on Google leads me to dozens of discussions on mailing lists, madwifi, hostap and Jean Tourrilhes's Linux Wireless page - which don't help me very much, because Senao is not mentioned anywhere.

But I have gotten some idea now of how wireless works on Linux.

The Senao website itself is not helpful. :(

A few more hours on Google lead to this excellent introduction to the Senao wireless cards by SeattleWireless - I learn that the Senao card is built around the Prism 2 chipset. They also say the HostAP drivers work.

Well, that's something that should get me started on getting the wireless card working...

And neat - I just found some detailed documentation of setting up a wireless card based on the Prism wireless card at skippy.net!

Made glibc 2.3 lrp package

I threw in the towel with uCLibc and created a glibc 2.3.2 LRP package. I also created LRP packages for libm and libpthread that are glibc based.

And hey - 7DS works on LEAF Bering-uCLibc/WRAP now! (Will try to add screenshots soon.)

The reason I decided not to use uCLibc was because:

1. I couldn't run it on the Linux box I was building under because I am not root - so no /lib/ld-uclibc.x.x (wonder why the links are hardcoded)
2. When I finally managed to transfer a uCLibc-compiled version of a simple program to the WRAP board, two things happened:
1. First, it complained that libgcc_x.so.1 was missing.
2. After I manually copied that library file, my program gave segmentation faults that I couldn't fix, because LEAF has no debugger.

So I went back and created my own glibc 2.3 LRP packages. I referred to the files that were in the glibc 2.2.5 package and created one - and it worked!

Compiling with uCLibc (and) CS MICE

I downloaded ucLibc's toolchain (apparently, you have to build a whole new toolchain called "buildroot" to compile programs with uClibc, but that wasn't too hard.)

Then I built it and compiled a simple program ... and the compile worked. Yay!

But the program wouldn't run because it was looking for /lib/ld-uclibc.x.x - which I couldn't install because I am not root. :(

There was something in uCLibc documentation about the paths being hardcoded, wonder why.

---------

Also, met with CU-CS webmaster Ben today - talked to him about how to use the MICE system. I will be hacking some PHP scripts to see if I can read data from MICE.

Eventually, this should develop into a Mambo component that can be plugged into the new Mambo version of the IRT website.

7DS on WRAP: ucLibc

My persistent efforts in making 7DS run on the LEAF/WRAP board hit a dead end last week.

The reason was that the latest glibc LRP package they have is 2.2.5 ... but my build requires 2.3!

I thought of creating an LRP package of glibc 2.3, but that may turn out to be a monster library!

My professor suggested last week that I reconsider and see if I can instead compile 7DS with ucLibc to get it running on WRAP, so I am taking a look at that...

Bering-ulibc installed on WRAP

Phew, after hours upon hours, I finally succeed in installing Bering-uLibC on the WRAP board!!!

First off, I couldn't just install and run it as I did with WISP ... Bering-ulibc is a different animal altogether.

There's a ton of documentation at this page: LEAF for the pcengines WRAP but some of it might need to be tweaked for your needs. What I did was basically as follows (on Windows XP machine with CF card reader):

1. Popped the CF into memory card reader.
   
2. Formatted using WinXP DOS format (format d: /FS:FAT)
   Note: CF needs to be formatted as FAT12 or FAT16, otherwise it won't boot. And don't worry about having to make a DOS 6.22 boot diskette; WinXP DOS's format works just fine - just make sure to specify /FS:FAT to format it as FAT16.
   
3. Download syslinux (http://syslinux.zytor.com/) and unzip the latest version on your drive.
4. Run syslinux -s d:
5. You should have already downloaded the latest Bering-ulibc package from the download website of the LEAF website.
6. Unzip the image and use WinImage to open the .imz file.
   Note: I wasn't able to unzip their .imz files using any software other than WinImage.
7. Unzip the Bering files to a local directory.
8. (All following instructions use the same directory.)
   
9. Download any additional packages and put them in the same directory
10. Edit leaf.cfg in this directory and make following changes:
    
    • Change /dev/fdxx to /dev/hda
      Note: The documentation says /dev/hda1, but I spent many hours trying to get that to work ... so when I changed it to /dev/hda, it finally worked. Perhaps WRAP uses /dev/hda because there is only one CF without partitions.
    • Append the additional packages to the LRP list so that they also boot up.
11. Edit syslinux.cfg and make the following changes:
    
    • Change /dev/fdxx to /dev/hda
12. Download the kernel with patch for WRAP that does not require a keyboard controller. Delete the old "linux" file and rename the newly downloaded "linux-2.4.x-xx.tux" to "linux".
13. Download the initrd-ide-cd file that will allow you to boot from the CF rather than a floppy. Delete the old "initrd" file and rename your "initrd-ide-cd" to "initrd".
14. Copy all your files to the CF.
    Note: You will notice you have an ldlinux.xx on your local directory as well as the CF. Leave the one on the CF intact - it was created by syslinux and you need that to boot the CF.

Remove the CF (after unmounting it first if needed), pop it into the WRAP board, and watch WRAP boot up with Bering-ulibc and your required packages!

Edit [July 14, 2005]: One step I missed mentioning here is that you have to make the serial port the primary means of input/output for communicating with the device. Here's some information from the LEAF documentation on how to use a serial port with LEAF.

In short, you have to edit syslinux.cfg (use 9600 baud, I had some trouble at higher speeds) and /etc/inittab. The /etc/inittab is the toughie, as you have to unpack the etc.lrp package, edit the file, repackage the LRP and copy it back to your CF.

India trip; Mozilla search

Today I have had to virtually cancel my India trip due to a lot of reasons. :( I don't expect to be going this year ... hopefully next year.

For the 7DS project, I have had to look at seeing how Mozilla's toolbar search works. It looks like it has to be built as a plugin for Mycroft. (Don't be scared by the word plugin - looks like simple "plugins" can be built using XML files.)

Another interesting thing is the keyword search from the address bar! Sounds interesting...

Other updates: WRAP is now going to use Bering-uclibc instead of WISP, since the Bering kernel is more recent than the WISP kernel, and also includes the wd1100.o driver by default, which should turn off the "keyboard jammed" messages.

I've had some trouble getting the Bering image to boot off my CF, but I think it's just a problem of not reading the documentation...

I'm sure there are other updates related to 7DS over the last week or so, but I've plain forgotten...

Getting the hang of LEAF

Whew, I am finally beginning to get the hang of LEAF, how to install packages, etc (at least theoritically.)

A lesson: for best documentation on open source projects, refer to the mailing lists, not the official website or documentation!

An aside: I've been looking at BREW and J2ME for porting our 7DS project to cell phones ... but it appears that J2ME beats BREW hands down? More on this to come.

sqlite time and date, garbage collection

Phew, I _finally_ managed to figure out how to garbage collect old values in a sqlite database.

There wasn't much documentation on the date/time functions in sqlite, so I had to do quite a bit of reading on that, strftime() as well as locale times (because that heavily influences the strftime () values.)

Anyway, here is the magic code I used:
DELETE FROM queries WHERE (strftime('%s', time_created) + keepalive*60 - strftime('%s','now', 'localtime') < 0)

where my table 'queries' has columns 'query', 'time_created' and 'keepalive' (in minutes).

The formula is obvious:
if (time_created + keepalive - current_time < 0) {
delete the query;
}

and that's what that SQL does.

Porting Linux to LEAF

Did I ever tell you how hard it is to port Linux applications to embedded systems?

Well, apparently to port your application to LEAF (running on WRAP with x86 and all), you still need to repackage all our libraries, binaries, etc into a LRP format (a highly compressed .tar.gz file).

So far, so good, but what happens when you have a ton of libraries to port to the WRAP? Like libxml, swish-e ... tinyHTTP?

And also, the documentation for making LRP packages is not that good, so you mostly have to find your way through it...

More on this to come.

leaf-wisp and 7DS queries

I removed m0n0wall from the WRAP box and "installed" a LEAF distro (branch actually) called "Wisp" which is a small version of LEAF meant for CF cards.

I just downloaded the binary and wrote it to the CF card, and it works perfectly - except for the silly and well-documented "pc_keyb: keyboard jammed" error message. Other than that, it seems to work smoothly - boots properly and gives me a shell prompt when I quit the configuration screen.

The 7DS query page is finished: see it in all its glory in HTML! (Page subject to change.)

Of course, it is far from perfect, and I still need to work more on tying the different pieces together and making them robust.

XML parsing and more on WRAP

I finally managed to get the XML parsing with libxml working to my heart's content. For one, using the -lxml apparently uses the old libxml libraries, so I tried compiling my programs with -lxml2 and that solved a lot of compiler and linker problems.

When I finally got XML working, my GNU time functions stopped working for some odd reason. gdb told me that my program was crashing on the localtime() call - while it had been working perfectly all along! Argh!

As for WRAP box updates - I found out last week that m0n0wall, neat as it is, has no shell and a very limited interface which only allows you to change IP address and a few other elementary settings.

Our idea of installing 7DS on m0n0wall on WRAP goes out the window!

But its sibling m0n0box seems interesting and more fully featured, though I would like to see if we can use Linux on WRAP ...

libxml & sqlite

Well, I finished sorting out sqlite and am able to work with it using C programs quite well. The callback functions are a little tricky, but I am getting the hang of them.

I also started using libxml today to encode and parse all those query and response messages. It's really neat and easy, but the code for parsing something is long and complicated! Despite that, working with XML seems to be fun...

More on sqlite and WRAP

Well, I finally managed to get sqlite working with my C program.

Installation was easy, and I did that last week. The C API was a little harder, but I got it working today.

I got the female-female null modem adapter and am going to use it to play with the m0n0box / WRAP system today.

NSF progress report and SQLite

Today I worked on (and finished) a NSF progress report for the components of the 7DS system that I worked on.

I also started playing with sqlite and reading through documentation for it. I need a database for my query-multicast system for 7DS, and it looks like this may be the right fit.

First post

This blog will be about my Computer Science research in the IRT Lab at Columbia University. I am hoping it will help others who struggle with complex hardware and software stuff... and also serve as a reminder for myself when I need to write documentation.