Two-way syncing Evolution Contacts and Calendar with Google on Ubuntu Trusty using SyncEvolution

SyncEvolution does the magic, this just is how to put it together.

Follow SyncEvolution install instructions for Debian GNU/Linux, Ubuntu

https://syncevolution.org/documentation/installation

Repeated here for convenience - do following as root or use sudo:

# echo "deb https://download.01.org/syncevolution/apt stable main" > /etc/apt/sources.list.d/syncevolution.list
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B2EC3981
# apt-get update
# apt-get install syncevolution-bundle

Now, as normal user in Evolution, create personal calendar and addressbook, both called "Google" (alternatively you can alter the database=Google below to whatever you want).

Then open a terminal and do:

$ syncevolution --configure --template google username=XXXXX@gmail.com password=YYYYY target-config@google
$ syncevolution --configure --template SyncEvolution_Client syncURL=local://@google database=Google username= password= google addressbook calendar
$ syncevolution --configure autoSync=1 autoSyncInterval=30M notifyLevel=0 target-config@google
$ syncevolution --configure autoSync=1 autoSyncInterval=30M notifyLevel=0 google
$ syncevolution --sync slow google

Where XXXXX is your gmail username and YYYYY is your password.
NOTE: username= password= is intentionally blank on second command.

To have this auto-started at login, do this:

$ cat > ~/.config/autostart/syncevo-dbus-server.desktop <<EOF
[Desktop Entry]
Type=Application
Hidden=false
Name=syncevo-dbus-server
Comment=SyncEvolution D-Bus Server
Exec=/usr/libexec/syncevo-dbus-server-startup.sh
EOF

Logout and back in to see if /usr/libexec/syncevo-dbus-server is running

Perfect Audio over RDP with padsp

Unfortunately, in the linux desktop world, some things just don't work.  Especially when it comes to webinars, web conferencing and the like, where the focus is on Windows desktops.  Using RDP to a virtual or physical Windows box is a good solution, except when it comes to audio.  Audio jitter/stutter is almost guaranteed on an RDP session.  Googling the topic, you'll even find several "experts" saying its just not possible to have quality audio over RDP.

The linux RDP client "rdesktop" was historically built against OSS, and under Ubuntu it was patched to work with ASLA, as OSS is kinda considered "old and non-standard".  So, yay, Ubuntu's rdesktop works with "-r sound:local" to get sound over an RDP session.. but again.. jittery - even over gigabit LAN.

Virtualbox comes with a custom RDP client, rdesktop-vrdp, which was tweaked for virtual rdp sessions, but still uses OSS, and therefore there is NO sound on RDP sessions (unless of course you build OSS support on your system - just not there in Ubuntu).

Alas!  The PulseAudio developers built this wrapper "padsp", which "starts  the  specified program and redirects its access to OSS compatible audio devices (/dev/dsp and auxiliary devices) to a PulseAudio sound server."  The result of running rdesktop-vrdp via this wrapper is just amazing.. audio is just PERFECT!  Absolutely no jitter in the audio at all.  Not only on a local network, but even across WAN.

Its so simple to launch too:  "padsp rdesktop-vrdp -r sound:local .....(whatever other params)"

padsp is part of pulseaudio-utils in Ubuntu

I have no idea why this wrapper works so good.. but kudos to the developers!  Or maybe its just OSS that is so efficient with networked audio??  If anyone knows, let us know in comments below.

Improving LTSP scripting facility

One thing that has been a pain with LTSP is that the scripts in /usr/share/ltsp/screen.d which control what launches on the thin client are all contained within the build image.  Any change/customization to those scripts then requires a rebuild/redeployment of the image.

I thought why not just use the tftp client to pull the script from the TFTP server instead.  Turns out this is quite easy to do.  I added the following script to the screen.d directory, and called it "tftp" (which was just based on one of the existing init scripts):

#!/bin/sh
#

#full path to file (excluding /var/lib/tftpboot)
filepath=$1
file=`basename $filepath`

if [ -n "$SERVER" ]; then
    script_temp=$(mktemp)
    tftp "$SERVER" -c get ${filepath} $script_temp
    # only execute if it has non-zero size.
    if [ -s "$script_temp" ]; then
        mv "$script_temp" /usr/share/ltsp/screen.d/${file}
        chmod 777 /usr/share/ltsp/screen.d/${file}
        exec /bin/bash /usr/share/ltsp/screen.d/${file}
    else
        rm "$script_temp"
        sleep 15
        exec /bin/bash --login
    fi
fi

Now in lts.conf I can do somthing like:

SCREEN_01 = tftp /ltsp-trusty/amd64/firefox

Now I have built the "/var/lib/tftpboot/ltsp-trusty/amd64/firefox" script on the TFTP server and it can easily/quickly customized as needed.

Here is an example (requires "apt-get install firefox openbox" in the ltsp-chroot image):

#!/bin/sh

. /usr/share/ltsp/screen-x-common

export HOME="${HOME:-/root}" USER="${USER:-root}"
COMMAND="openbox-session"
mkdir -p /root/.config/openbox
echo "/usr/bin/firefox http://somesite.biz" >> /root/.config/openbox/autostart
chmod 755 /root/.config/openbox/autostart

# The following logic is described at the top of xinitrc.
if [ -x /usr/share/ltsp/xinitrc ]; then
    xinit /usr/share/ltsp/xinitrc "$COMMAND" -- "$DISPLAY" "vt${TTY}" $X_ARGS >/dev/null
else
    eval "xinit $COMMAND -- $DISPLAY vt${TTY} $X_ARGS >/dev/null"
fi

if [ $? -ne 0 ]; then
    echo "Xserver failed, falling back to a text shell" >&2
    exec /bin/bash --login
fi

So simple things like having one thin-client launch firefox and another launch NX player, is easy thru lts.conf and script manipulation - no rebuilding of images required (apart from package installation of course).

Ubuntu LTSP for the Enterprise

Ubuntu LTSP (ltsp-server package) provides a very easy and convenient method for deploying a thin-client environment (see ltsp.org), but is centered primarily around one server being the DHCP/TFTP/NBD and image build server.  Being in an enterprise environment, I needed to be able to split these functions in order to deploy thin clients.  The image below depicts what I needed to achieve. Essentially I wanted to have ubuntu 14.04 images served by older release servers, so my "build" server needs to be 14.04.  Plus I wanted to be able to server multiple custom images off the same server(s).  Note:  I have existing DHCP/TFTP and NBD servers - so this post assumes these are already established - If you do not, then the ltsp-server package has everything you need, and this post is kinda moot.

First, on my build server, I did the following:

apt-get install ltsp-server
ltsp-build-client --dist trusty --arch amd64 --base /opt/ltsp-trusty

This builds your base client system, in my case an amd64 architecture ubuntu 14.04 base.  It takes about 20 minutes depending on your Internet connection and system speed.  Then any customizations can be done via ltsp-chroot:

ltsp-chroot -a amd64 -b /opt/ltsp-trusty --mount-package-cache

(more on what I customized in later post)

And to build the image:

ltsp-update-image --base /opt/ltsp-trusty

So what we have now on the build server, is the image in /opt/ltsp-trusty/images called amd64.img and the needed tftpboot files in /var/lib/tftpboot/ltsp.  In very simple terms, we now need to deploy the image "amd64.img" to the NBD server, and the PXE boot environment to the TFTP server, and then tweak configuration files to point to the right places.

NBD server (Network Block Device)

For every different image you want to deploy, you need nbd-server listening on a different port.  I am using inetd to spawn the nbd-server instances.  Here is my inetd.conf, which is configured for 3 images on ports 2000 thru 2002:


2000  stream  tcp nowait  nobody /usr/sbin/tcpd /usr/sbin/nbdrootd /opt/ltsp/images/i386.img
2001  stream  tcp nowait  nobody /usr/sbin/tcpd /usr/sbin/nbdrootd /opt/ltsp-trusty/images/amd64.img
2002  stream  tcp nowait  nobody /usr/sbin/tcpd /usr/sbin/nbdrootd /opt/ltsp-trusty/images/i386.img

The image files have to be accessible directly, so in my case I copied them from my build server.

TFTP server (Trivial File Transfer Protocol)

For every different image, you need a specific PXE environment defined in /var/lib/tftpboot.  In my case I have 3:

/var/lib/tftpboot/ltsp/i386
/var/lib/tftpboot/ltsp-trusty/amd64
/var/lib/tftpboot/ltsp-trusty/i386

These can be copied as is from the same location on the build server - each environment can have its own lts.conf file.

Now, there is some tweaking required in order for the client to grab the correct NBD image.  You need to add nbdroot={ipaddress}:{port} to the kernel boot parameters for any image served by the non-standard 2000 NBD port.  This is done in the pxelinux.cfg/ltsp file - snippet below.

# This file is regenerated when update-kernels runs.
# Do not edit, see /etc/ltsp/update-kernels.conf instead.
default ltsp-NBD
ontimeout ltsp-NBD

# This file is regenerated when update-kernels runs.
# Do not edit, see /etc/ltsp/update-kernels.conf instead.
label ltsp-NBD
menu label LTSP, using NBD
kernel vmlinuz-3.13.0-37-generic
#append ro initrd=initrd.img-3.13.0-37-generic init=/sbin/init-ltsp quiet splash root=/dev/nbd0
append ro initrd=initrd.img-3.13.0-37-generic init=/sbin/init-ltsp nosplash root=/dev/nbd0 nbdroot=192.168.202.4:2001
ipappend 2

DHCP server (Dynamic Host Configuration Protocol)

Your DHCP configuration is key, as it directs the PXE booting client to the correct TFTP server and PXE path.  I use an Etherboot gPXE image served by a local TFTP server to fix issues with older network cards - but you may not need that.  Here is one client config section example:

host webc2 {     hardware ethernet 20:cf:30:6f:0a:9e;

     fixed-address webc2;

     option host-name "webc2";

     if substring (option vendor-class-identifier, 0, 9) = "PXEClient"           {

   if exists user-class and option user-class = "gPXE" {

      next-server 192.168.202.4;

      filename "/ltsp-trusty/i386/pxelinux.0";

   } else { filename "/tftpboot/undionly.kpxe"; }

} else {

  next-server 192.168.202.4;

  filename "/ltsp-trusty/i386/nbi.img";}}

The "next-server" portion points it to the TFTP server and PXE file.

You'll probably notice that in my case, my TFTP server and NBD server are on the same server, but its not necessary.

Anyway, in another post I'll discuss an improvement I made to LTSP to allow better scripting via TFTP.

Inserting/Pasting HTML into OpenOffice Calc via Basic macro

Its easy to manually copy data from a browser window and paste into an existing OO Calc doc.  If tables are used in the html, they paste nicely into rows and columns.

I needed to automate this process via a macro, but it wasn't as simple as I at first thought.

Its easy to use a filter with loadComponentFromURL, to create a NEW Calc doc.  But I wanted to place it specifically into an existing doc.

I then found ability to link to External Data via macro using AreaLinks.insertAtPosition.  This worked wonderfully, but I didn't like that it retained links.. I just wanted a dump of the data, no strings attached.

I moved to try doing copy and paste via a Shell command, wget | xclip, and .uno:Paste , but the paste was always "unformatted text" and thus pasted raw html.

I went back to AreaLinks.insertAtPosition .. figured if I can insert a link, and then remove the link (retaining the data), I've essentially achieved my objective.  So this is the macro I ended up with (works with OO 3.2 and LO 4.1):

Sub InsertWebLinkAtSelection(url As String, section As String)

   Dim oDoc As Object

   Dim oSelection

   Dim oAddress As new com.sun.star.table.CellAddress

      oDoc = ThisComponent

   oSelection = oDoc.getCurrentController().getSelection()

   if oSelection.supportsService("com.sun.star.sheet.SheetCell") then

      oAddress = oSelection.getCellAddress()

      oDoc.AreaLinks.insertAtPosition(oAddress,url,"HTML_" +

section,"calc_HTML_WebQuery","0 0")

      oDoc.AreaLinks.removeByIndex(getCount())

   endif

End Sub

The key was that last line... oDoc.AreaLinks.removeByIndex(getCount())  .. doesn't affect any exist links in document.  So it creates a link, which inserts the data at current cell address, and then removes the link.

Note:  The "section" argument is either "all" or "1", "2", "3",.. etc representing which html table element you want.

Note 2:  I also found that this method doesn't respect the http://user@pw:URL method, so I did have to use : Shell ("bash",0,"-c 'wget -O /tmp/file.html http://user@pw:URL....'", true) to get the results to a temp file, and then used InsertWebLinkAtSelection("file:///tmp/file.html","all")

MarkLogic - using xquery to interact via Alfresco's content repository using the REST API

Using Alfresco community edition 4.2.a, and MarkLogic 6.

It was difficult navigating Alfresco's documentation on the REST API, as there were NO examples, just specs.. and often the specs stated that they were "deprecated".  Ultimately I found this document on CMIS and interacting via http.  (NOTE:  There is a vital typo on page 25 in the cmisra namespace declaration, which causes example to fail.. just remove extraneous slash).

Anyway, I cobbled together an example that demonstrates login, ticket validation, content creation, and logout.  This works in the Query Console.. although the base64 module is my own (see prior post on base64 encoding of binary files).

xquery version "1.0-ml";
declare namespace html = "http://www.w3.org/1999/xhtml";
import module namespace json = "http://marklogic.com/xdmp/json" at "/MarkLogic/json/json.xqy";
import module namespace base64 = "base64" at "utility/base64.xqm";


let $string := concat('{ "username":"admin", "password":"admin" }')
let $data := <data>{$string}</data>

let $login := xdmp:http-post("http:/localhost:8080/alfresco/service/api/login",
  <options xmlns="xdmp:http">
        <data>{$string}</data>
        <headers>
            <content-type>application/json</content-type>
        </headers>
    </options>)
    
let $ticket_xml := json:transform-from-json($login[2])

let $ticket_id := data($ticket_xml//*[local-name() = 'ticket'])

let $uri := concat("http://localhost:8080/alfresco/service/api/login/ticket/",$ticket_id,"?alf_ticket=",$ticket_id)

let $check := xdmp:http-get($uri)
(: if you have a node_id, the following is example of how to retrieve content
let $content_uri := concat("http://localhost:8080/alfresco/service/cmis/i/", $node_id, "/content?alf_ticket=", $ticket_id) :)

let $filedata := base64:encode("/some/path/image.png")  (: insert your own base64 here :)

let $content_uri := concat("http://localhost:8080/alfresco/s/cmis/p/path/children", "?alf_ticket=", $ticket_id)  (: the "path" is the path after "Company Home" :)

let $atomxml :=
<entry xmlns="http://www.w3.org/2005/Atom" 
xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908" 
xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908">
 <title>image.png</title>
 <summary>A sample upload</summary>
 <content type="image/png">{$filedata}</content>
 <cmisra:object>
 <cmis:properties>
 <cmis:propertyId propertyDefinitionId="cm:description"><cmis:value>This is the description property</cmis:value></cmis:propertyId>
 <cmis:propertyId propertyDefinitionId="cmis:objectTypeId"><cmis:value>cmis:content</cmis:value></cmis:propertyId>
 </cmis:properties>
 </cmisra:object>
</entry>

let $upload := xdmp:http-post($content_uri,
  <options xmlns="xdmp:http">
        <data>{concat('<?xml version="1.0" encoding="utf-8"?>',xdmp:quote($atomxml))}</data>
        <headers>
            <content-type>application/atom+xml</content-type>
        </headers>
    </options>)

let $logout := xdmp:http-delete($uri)

return ($check,$upload)

MarkLogic base64 encoding of a binary file using MLJAM

MarkLogic has the function xdmp:base64-encode, which unfortunately only works on xs:string's.

I needed a function to convert an arbitrary file (ie. binary) into a base64 encoded stream in order to put it into an atom+xml http post to feed Alfresco document management system.  Luckily I found MLJAM, which allows you to embed Java into xquery.  I found the needed Java snippet here, and then put it together - the code below should work in the MarkLogic Query console, just set the filename parameter accordingly.

Ultimately MarkLogic should have xdmp:base64-encode work on a binary node, then you could simply do xdmp:base64-encode(xdmp:external-binary("/some/path/image.png"))!

xquery version "1.0-ml";

import module namespace jam = "http://xqdev.com/jam" at "/MarkLogic/jam.xqy";

let $code := <code><![CDATA[

import org.apache.commons.codec.binary.Base64;

private String encodeFileToBase64Binary(String fileName)

throws IOException {

File file = new File(fileName);

byte[] bytes = loadFile(file);

byte[] encoded = Base64.encodeBase64(bytes);

String encodedString = new String(encoded);

return encodedString;

}

 

private static byte[] loadFile(File file) throws IOException {

   InputStream is = new FileInputStream(file);

     long length = file.length();

   if (length > Integer.MAX_VALUE) {

       // File is too large

   }

   byte[] bytes = new byte[(int)length];

   

   int offset = 0;

   int numRead = 0;

   while (offset < bytes.length

          && (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {

       offset += numRead;

   }

 

   if (offset < bytes.length) {

       throw new IOException("Could not completely read file "+file.getName());

   }

 

   is.close();

   return bytes;

}

  

    return encodeFileToBase64Binary(filename);

]]></code>

  

let $path := "http://localhost:8080/mljam/mljam"

let $connect := jam:start($path, (), ())

let $vars := jam:set("filename", "/some/path/image.png")

let $eval := jam:eval-get($code)

let $end := jam:end()

return $eval

League of Legends on 64-bit ubuntu 12.04 with Intel graphics

With the release of wine 1.6, LOL works well under linux, but was having black background issue.

Found this link: https://wiki.archlinux.org/index.php/Intel_Graphics#Black_textures_in_video_games

Installed driconf and enabled S3TC texture compression support.. this solved the problem.

IMAP setup on BB10 (Z10)

Its soo simple now.. way to go BB.

First - "Add Account"

Click on "Advanced"



















Click on "IMAP"




















Enter details.  Custom ports supported..  PUSH supported for sure with Cyrus imap server.  Very nice.. no third-party app required.

Installing Ubuntu 13.04 on an ASUS X55A / X55U

Downloaded the official release of 13.04 yesterday, and tried installing on an ASUS X55A.  I wanted a dual-boot setup.  Took a little bit of effort, but got er done.. steps follow.

First thing I had to do was to disable Fast Boot and disable Secure Boot.  From inital power on, press ESC repeatedly until small blue boot menu appears, then select "Enter Setup".  Goto Boot -> Boot Configuration and disable Fast Boot. Change boot order so that DVD is 1st.  Goto Security and disable Secure Boot Control.  Press F10 to save and exit.  This allows the install media to boot (in my case a DVD).  Install went fine, selected the "Install Ubuntu alongside Windows 8" option.

After install, the system booted to grub, but when selecting Ubuntu, screen would go blank (but not off) and all HDD activity would stop.  Numlock/Capslock keys worked, so system was not hung.  Pressing Enter a few times would cause HDD light to flicker, but otherwise nothing.. Ctrl-Alt-Fn would not do anything.  Strange thing was that if I rebooted, pressed ESC to get blue boot menu, and selected the "ubuntu" option (created by installer), it would goto grub and then boot fine!

To fix this, I installed and ran boot-repair.

sudo add-apt-repository ppa:yannubuntu/boot-repair && sudo apt-get update

 
sudo apt-get install -y boot-repair && (sudo boot-repair &)

I just selected the "recommended" option in boot-repair.  After doing this, I had to add another boot option as per the instructions - had to add sda1/EFI/ubuntu/grubx64.efi.  Pay attention to what it tells you, as it may not be identical, depending on your partitioning.

Reboot and press ESC on boot to get blue menu, Enter Setup, goto Boot -> Add New Boot Option -> Add boot option and give it a name (i used "ubuntu13.04").  The filesystem was pre-selected for me, the path for the boot option I entered as follows:  fsx:sda1/EFI/ubuntu/grubx64.efi  - have no idea what the fsx: is about, but it was given in the help text.  Scroll down to Create, and press enter.  Press ESC to go back, and then move the newly created boot option to #1 position.  Reboot and enjoy.  Fully functional dual-boot system with Win8 and Ubuntu 13.04.

Why the Ubuntu folks don't learn from the boot-repair folks is really a mystery to me...

NOTE:  I ended up having to delete the Windows 8 boot option in the BIOS, because after booting from grub into Win8 once, it did something to bypass grub.. frikin Windows!

Upgrading RT 3.6.4 to RT 4.0.10

Request-Tracker (RT) upgrade steps.  This is based on http://blog.bestpractical.com/2011/07/upgrading-to-rt-4.html.  Please read all related documents and don't just follow this blindly.

Moving from a gentoo linux 2.4 box, RT 3.6.4, mysql 4.0.8 to an ubuntu 12.04 server, RT 4.0.10 and mysql 5.5

Export 3.6.4 Database
On RT 3.6.4 machine as root (obviously must know the root mysql password):

mysqldump -p --opt rt3 > rt3.sql

Copy this file to the new machine.  All further steps are on new machine.

Prepare SQL for Import
You have to edit rt3.sql with vim or sed, and remove all references to "Type=Innodb" (this is default in mysql now), and search for "timestamp(" and remove the bracketed section.. (eg. "timestamp(14)" is now just "timestamp").

Create Database for Import

mysql -p
create database rt4 character set latin1;
grant all on rt4.* to rt_user identified by 'rt_pass';
quit

NOTE: above replace rt_user and rt_pass by the user/password you wish to use for RT.  Also, it may seem counter intuitive to name database rt4, since we are importing rt3 - just remember the end target is 4.0.10.

Import 3.6.4 Database

mysql -p --default-character-set=binary rt4 < rt3.sql

Download/Configure/Install RT 4.0.10

Download RT 4.0.10 source, extract to temp dir, run .configure with needed parameters (see your 3.6.4 install), and basically follow README up to and including "make install".  This should install in /opt/rt4.  Be sure to configure RT_SiteConfig.pm and configure apache2 as needed (beyond scope of this post).

Upgrade Database
From the temp dir used above execute the following:

make upgrade-database
(From 3.6.4 to 3.7.87)

If it fails at 3.7.81, do this:
mysql -p rt4
CREATE INDEX CachedGroupMembers3 on CachedGroupMembers (MemberId, ImmediateParentId);
quit
make upgrade-database
(From 3.7.81 to 3.7.87)

perl etc/upgrade/upgrade-mysql-schema.pl rt4 rt_user rt_pass > queries.sql
(again replacing rt_user and rt_pass with user/password you wish to use for RT)
mysql -p rt4 < queries.sql
(this takes about 3 times as long as the initial import!)
mysql -p
alter database rt4 default character set utf8;
quit
make upgrade-database
(From 3.7.87 to 4.0.9)

If it fails at 3.8.2, do this:
mysql -p rt4
drop index groupmembers1 on GroupMembers;
create unique index groupmembers1 on GroupMembers (GroupId,MemberId);
quit
make upgrade-database
(From 3.8.3 to 4.0.9)

perl -I /opt/rt4/local/lib -I /opt/rt4/lib etc/upgrade/shrink_cgm_table.pl
(takes even longer than above query)
perl -I /opt/rt4/local/lib -I /opt/rt4/lib etc/upgrade/shrink_transactions_table.pl
perl etc/upgrade/vulnerable-passwords --fix

Now test, test, test... :)

PXE booting of Webconverger

Webconverger is a great web kiosk tool.  This post briefly explains how to customize it and have it boot over a network (PXE boot).  These instructions apply to Ubuntu servers, but concept is adaptable.

First, download the webconverger ISO - I used version 8.0 since I already had it kicking around.

You need a DHCP server, a TFTP server and an NFS server.  Of course, the workstation has to support PXE booting, and be on the same network as these 3 servers.

You can use one physical machine for all three services, but in my scenario I have two machines, one acts as DHCP server, and the other is both the TFTP and NFS server.

DHCP

Using dhcp3-server.. entry in /etc/dhcp3/dhcpd.conf:

        host temp {
                hardware ethernet 00:14:c2:c4:5f:38;
                fixed-address temp.abc.com;
                option host-name "temp";
                option root-path "192.168.202.4:/tftpboot/webc/";
                if substring (option vendor-class-identifier, 0, 9) = "PXEClient" {
                next-server 192.168.202.4;
                filename "webc/pxelinux.0"; }
        }

 Note:  Here the "next-server" is required to redirect to the TFTP server.. obviously change the MAC and ip addresses according to your workstation and servers.  Be sure to restart your DHCP server.

Ok - thats the easy part..  now you have to prep for the TFTP/NFS server setup:


For PXE booting, you need syslinux, nfs-kernel-server, and tftpd-hpa.

TFTP

First, make root tftpboot directory for webconverger, then mount the ISO and copy files into the root dir:

mkdir -p /var/lib/tftpboot/webc/webconverger/8.0
mount -o loop webc8.iso /mnt
cp -a /mnt/* /var/lib/tftpboot/webc/webconverger/8.0
umount /mnt
cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/webc
cp /usr/lib/syslinux/vesamenu.c32 /var/lib/tftpboot/webc
mkdir /var/lib/tftpboot/webc/pxelinux.cfg

In /var/lib/tftpboot/webc/pxelinux.cfg create a file called "default" with the following contents:

DEFAULT vesamenu.c32
TIMEOUT 10
ONTIMEOUT wc
PROMPT 0
NOESCAPE 1
MENU BEGIN Webconverger
MENU TITLE Webconverger
        MENU INCLUDE webconverger/webconverger.menu
MENU END

NOTE:  The TIMEOUT 10 is basically allowing no time to show menu.. I just want it booting "wc" entry (ONTIMEOUT wc) - adjust accordingly.

Last thing needed is the file /var/lib/tftpboot/webc/webconverger/webconverger.menu:

# nfsroot ip is pxe server's address

LABEL wc
        MENU LABEL Webconverger 8
        KERNEL /webconverger/8.0/live/vmlinuz
APPEND initrd=/webconverger/8.0/live/initrd.img boot=live config homepage=http://www.google.com hostname=webconverger nonetworking nopersistent quickreboot nomodeset noroot splash username=webc vga=current quiet netboot=nfs nfsroot=192.168.202.4:/var/lib/tftpboot/webc/webconverger/8.0
        TEXT HELP
        Boot Webconverger 8.0
ENDTEXT

NFS

Essentially you just need one entry /etc/exports:

/var/lib/tftpboot/webc *(ro,sync,no_subtree_check)

Then do:  exportfs -a

Workstation

Enter BIOS and configure station to boot from network.

You can customize by workstation, by creating additional entries in pxelinux.cfg..    a file called "01-00-14-c2-c4-5f-38" for example, will be used for workstation with MAC of 00:14:c2:c4:5f:38.  Essentially you just copy the "default" file and adjust the ONTIMEOUT to point to a LABEL in webconverger.menu.

 

MarkLogic xdmp:email with binary attachments need wrapping

According to RFC 2045, MIME base64 encoded attachments need to be wrapped at 76 characters.

Most email clients can handle attachments that are not wrapped (meaning the base64 encoding is not wrapped at 76 charaters), but some, specifically MS Outlook (at least some versions) cannot, and report that the attachment is corrupt.

When sending binary attachments from MarkLogic via xdmp:email
, this wrapping does not occur - causing some recipients to report documents are corrupt.

In order to fix this, we created a function which wraps the base64 encoding prior to sending.

(: Function used for wrapping email attachments :)
declare function local:wrap($wrapped, $unwrapped, $max){
        let $remaining := string-length($unwrapped)

        return if ($remaining <= $max) then concat($wrapped, "&#13;&#10;", substring($unwrapped, 1, $max))
        else local:wrap(concat($wrapped, "&#13;&#10;", substring($unwrapped, 1, $max)), substring($unwrapped, $max+1, $remaining), $max)
};

And it used when building the parameter list:

        let $attachment1 := local:wrap("", xs:string(xs:base64Binary(doc("/some/doc/path/doc.pdf"))), 76)

See documentation on xdmp:email for complete example with attachments.

Linux memory tweaks for improved responsiveness

Memory management techniques in the linux kernel has changed over the years.  Sometimes a kernel upgrade, or a hardware upgrade, where you expect better performance, can lead to disappointment.

For a while now, I've been struggling with our ubuntu 10.04 application server (managing 35 desktop sessions) where we experience "morning sluggishness" with the system.  Between 6am and noonish, the system was just not as responsive as it could be, but in the afternoons it became quite zippy.  I've been explaining this away with things like "everyone is just logging in morning, system is seeing a heavy load early in the day".. or "everyone does more work in the morning, and take it easy after lunch".

I decided to analyze memory usage patterns combined with I/O wait times.  Between 8am and noon, there was an average I/O wait time of between 2% - 3%, and between noon and 4pm an average of < 1%.  When I graphed out memory usage patterns, I noticed that our nightly backups were consuming a huge amount of buffered memory (understandably).. and that this buffer was slowly reclaimed overtime and around noonish every day it got close to cache size.  This couldn't be coincidence.. performance steadily increased as the buffers from previous nights backup were free'd up.  Makes total sense.. but question was.. how do I modify this behaviour?

Googling around, I came across this link

I already knew about vm.swappiness - already have this set to 10.

But vm.vfs_cache_pressure and vm.drop_caches were new to me.  So I thought, if we can drop the caches right after the backup, and give a little more priority to the inode/dentry caches - this should reduce our I/O wait.

So, I set vm.vfs_cache_pressure to 50 ( sysctl -w vm.vfs_cache_pressure=50 ), and in my backup script, added "echo 3 > /proc/sys/vm/drop_caches" to the end.  The results are quite dramatic.  So far this morning, I have an average I/O wait of  0.19%, down from 2-3%!! (and system is very zippy.)  This graph gives a good visual on the memory usage...

Notice: Everyday at 00:00.. this is backup kicking in.. Tues/Wed shows the slow reclaiming of buffers.  Thurs is where I drop the caches right after backup.. now caches reach previous days peak by 8am!!

Could this be the end of my users complaining about a slow system??   Doubt it... They never notice when its fast.. but always and only notice when its slow.

Update:  After two days now.. amazing.. average daytime I/O wait is now < 0.10%

WinTV 1600 model 1199 and VirtualBox on linux

Had a working VirtualBox WinXP guest on an Ubuntu Natty host.

Installed a WinTV 1600 video capture card, and after that, the VirtualBox WinXP guest would not boot.. got the message:

Failed to load VMMR0.r0 VERR_NO_MEMORY

After digging thru the logs, and noticing vmalloc errors - I decided to blacklist the cx18 modules (for capture card), and reboot. VirtualBox worked now, but then if I removed the blacklist on the cx18 modules, and tried to load the module - I got the same vmalloc errors.

Determined its a shortage of virtual memory in the linux kernel. Default is 128M.

Certain device drivers (modules) - in my case, a video capture card - require chunks of this virtual memory for device mapping.

Virtualbox obviously uses the same mechanism. You can have 16GB of RAM, but if you only have 128M of vm allocated, you are going to run into the same issue.

Solution is to pass the "vmalloc=<size>" parameter to your kernel on boot.  Preferably in 64M chunks.

In ubuntu, edit /etc/default/grub, and add "vmalloc=192M" to the GRUB_CMDLINE_LINUX_DEFAULT list.

eg. GRUB_CMDLINE_LINUX_DEFAULT="vmalloc=192M quiet splash"

sudo update-grub

reboot

Now they play nicely together - and the overall system performance is much snappier.

Using curl with eXist to perform XQuery

Simple trick to use HTTP POST with curl to do an XQuery - (I struggled for a bit with the Content-type)

curl --header "Content-type: text/xml" --data "@x.xml" http://hostname:8080/exist/rest/db/

where x.xml is something like:


<?xml version="1.0" encoding="UTF-8"?>
<query xmlns="http://exist.sourceforge.net/NS/exist" start="1" max="20">
<text>
  <![CDATA[
  some XQuery here...
  ]]>
</text>
<properties>
  <property name="indent" value="yes"/>
</properties>
</query>

ext4 performance and the barrier option

We recently implemented a 6-disk SAS RAID 1+0 /home ext4 filesystem on our application server to improve performance, and found that we still had a disk I/O bottleneck, with processor I/O wait averaging in the 3 to 4% range.

I came across this LWN article http://lwn.net/Articles/283161/ that talks about a 30% performance drop when the barrier option is enabled on an ext4 filesystem.

As of the time of the LWN article, barrier was not enabled by default on an ext4 filesystem.  But somewhere along the kernel release line, barrier=1 (on) became the default for newly created ext4 filesystems.

Doing a "cat /proc/mounts" on my ubuntu 10.04 distro, I found that my ext4 filesystems indeed had the option barrier=1. So on April 3rd, I added barrier=0 to my fstab, and rebooted.  The performance boost was quite dramatic.

Using measures of "average wait time", "% utilization" and "%IO wait", here is the before and after stats, measured and averaged on a typical work week 8am to 5pm...


With barrier=1:

Average wait time:   90 ms
Average Disk utilization %:   40%
Average CPU IO wait %:  3.5%

With barrier=0:

Average wait time:  2 ms
Average Disk utilization %:   2%
Average CPU IO wait %:  0.40%


Graphs depict the change more dramatically (comparing April 1st to April 4th):

Average Wait Time

Average % Utilization

Load Average

NOTE: This is not a "very busy" April 1st compared to a "very light" April 4th.  Both are with a 30 user load, approx. 2300 processes.  And every other day of the week in the comparison show the same dramatic improvement.

Of course one can argue that we are sacrificing data integrity for performance - but we do have a UPS backed system, and crashes are (thus far) non-existent.

Finding the WWN on an HS22 SAS connectivity module

Recently obtained an IBM HS22 blade server with a SAS connectivity module, and was attempting to map a RAID volume in a disk storage module to this blade..  when I realized the AMM did not report the SAS ID (WWN) as it normally does with my HS21 blades.

I figured it was a firmware issue, so I upgraded the AMM firmware - but still nothing.

I found this link:http://www-947.ibm.com/support/entry/portal/docdisplay?brand=5000020&lndocid=MIGR-5083823  which scared me - seemed to indicate I had the wrong hardware.

I poked around the LSI BIOS and noted every SAS ID I could find.. and tried using those to map the volume, all to no avail.

It didn't help that the AMM reported the SAS module part number as 49Y8009, whereas I had ordered 43W4068.. thought there was a shipping error or something, but it turns out the 49Y8009 is just the internal IBM part number.

Thinking it was still related to firmware.. I upgraded the uEFI and the IMM firmware on the HS22.  Again, this helped nothing.

In a desperate final attempt, I decided to delve into the SAS RAID controller itself..  accessed the web interface, selected "Monitor SAS Module", and voila!  There, next to "Blade Slot Connection 2" was a SAS ID that was never presented anywhere else.  I mapped the volume using this WWN, and now all is good.

What a nightmare to find though..   found forum posts with no answers:  https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14418964&#14418964

Someone from IBM, please update your documentation!!

Aggregating row data using user-defined aggregate in Informix IDS

Implementing mysql group_concat() aggregate in Informix IDS 11.5 - thanks to Jonathan Leffler.

As DBA do:

CREATE FUNCTION gc_init(dummy varchar(255)) RETURNING LVARCHAR;
    RETURN '';
END FUNCTION;

CREATE FUNCTION gc_iter(result LVARCHAR, value VARCHAR(255))
    RETURNING LVARCHAR;
    IF result = '' THEN
        RETURN TRIM(value);
    ELSE
        RETURN result || ',' || TRIM(value);
    END IF;
END FUNCTION;

CREATE FUNCTION gc_comb(partial1 LVARCHAR, partial2 LVARCHAR)
    RETURNING LVARCHAR;
    RETURN partial1 || ',' || partial2;
END FUNCTION;

CREATE FUNCTION gc_fini(final LVARCHAR) RETURNING LVARCHAR;
    RETURN final;
END FUNCTION;

CREATE AGGREGATE group_concat
    WITH (INIT = gc_init, ITER = gc_iter,
          COMBINE = gc_comb, FINAL = gc_fini);

Then you need to grant access to the functions to users that need it:

grant execute on function gc_init to appsql;
grant execute on function gc_iter to appsql;
grant execute on function gc_comb to appsql;
grant execute on function gc_fini to appsql;


Example Use:

select ord_id, group_concat(product)
from ord
group by ord_id

xscreensaver and empathy presence notification

xscreensaver doesn't support d-bus - and therefore when screensaver activates, empathy (IM client) doesn't change status to "Away", like it does with gnome-screensaver.

The xscreensaver man page includes a nice perl script which you can use to accomplish the same thing.  So here is "watchx.pl" script:

#!/usr/bin/perl

my $blanked = 0;
 open (IN, "xscreensaver-command -watch |");
 while () {
     if (m/^(BLANK|LOCK)/) {
         if (!$blanked) {
             system "/usr/local/bin/empathy-away.sh";
             $blanked = 1;
         }
     } elsif (m/^UNBLANK/) {
         system "/usr/local/bin/empathy-avail.sh";
         $blanked = 0;
     }
 }

Then, using dbus-send, you can set status in empathy.. I use the following scripts "empathy-away.sh":

#!/bin/bash
USER=`whoami`
service=`qdbus | grep $USER | sed -e "s/ //g"`
path=`echo $service | sed -e "s/\./\//g"`
dbus-send --dest=$service /$path org.freedesktop.Telepathy.Connection.Interface.SimplePresence.SetPresence string:"away" string:"Away from keyboard"

and "empathy-avail.sh":

#!/bin/bash
USER=`whoami`
service=`qdbus | grep $USER | sed -e "s/ //g"`
path=`echo $service | sed -e "s/\./\//g"`
dbus-send --dest=$service /$path org.freedesktop.Telepathy.Connection.Interface.SimplePresence.SetPresence string:"available" string:"Available"

NOTE:  This is designed for one empathy account.. and that account name containing the users login name.  Works on ubuntu lucid.  Also - that dbus-send command is all on one line.

And to have it launch at login... create the following "watchx.desktop" file and place it in /etc/xdg/autostart:

[Desktop Entry]
Type=Application
Exec=/usr/local/bin/watchx.pl
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
Name[en_US]=Watch screen saver
Name=Watch screen saver
Comment[en_US]=Watch screen saver
Comment=Watch screen saver