Two-way syncing Evolution Contacts and Calendar with Google on Ubuntu Trusty using SyncEvolution

SyncEvolution does the magic, this just is how to put it together.

Follow SyncEvolution install instructions for Debian GNU/Linux, Ubuntu

https://syncevolution.org/documentation/installation

Repeated here for convenience - do following as root or use sudo:

# echo "deb https://download.01.org/syncevolution/apt stable main" > /etc/apt/sources.list.d/syncevolution.list
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B2EC3981
# apt-get update
# apt-get install syncevolution-bundle

Now, as normal user in Evolution, create personal calendar and addressbook, both called "Google" (alternatively you can alter the database=Google below to whatever you want).

Then open a terminal and do:

$ syncevolution --configure --template google username=XXXXX@gmail.com password=YYYYY target-config@google
$ syncevolution --configure --template SyncEvolution_Client syncURL=local://@google database=Google username= password= google addressbook calendar
$ syncevolution --configure autoSync=1 autoSyncInterval=30M notifyLevel=0 target-config@google
$ syncevolution --configure autoSync=1 autoSyncInterval=30M notifyLevel=0 google
$ syncevolution --sync slow google

Where XXXXX is your gmail username and YYYYY is your password.
NOTE: username= password= is intentionally blank on second command.

To have this auto-started at login, do this:

$ cat > ~/.config/autostart/syncevo-dbus-server.desktop <<EOF
[Desktop Entry]
Type=Application
Hidden=false
Name=syncevo-dbus-server
Comment=SyncEvolution D-Bus Server
Exec=/usr/libexec/syncevo-dbus-server-startup.sh
EOF

Logout and back in to see if /usr/libexec/syncevo-dbus-server is running

Perfect Audio over RDP with padsp

Unfortunately, in the linux desktop world, some things just don't work.  Especially when it comes to webinars, web conferencing and the like, where the focus is on Windows desktops.  Using RDP to a virtual or physical Windows box is a good solution, except when it comes to audio.  Audio jitter/stutter is almost guaranteed on an RDP session.  Googling the topic, you'll even find several "experts" saying its just not possible to have quality audio over RDP.

The linux RDP client "rdesktop" was historically built against OSS, and under Ubuntu it was patched to work with ASLA, as OSS is kinda considered "old and non-standard".  So, yay, Ubuntu's rdesktop works with "-r sound:local" to get sound over an RDP session.. but again.. jittery - even over gigabit LAN.

Virtualbox comes with a custom RDP client, rdesktop-vrdp, which was tweaked for virtual rdp sessions, but still uses OSS, and therefore there is NO sound on RDP sessions (unless of course you build OSS support on your system - just not there in Ubuntu).

Alas!  The PulseAudio developers built this wrapper "padsp", which "starts  the  specified program and redirects its access to OSS compatible audio devices (/dev/dsp and auxiliary devices) to a PulseAudio sound server."  The result of running rdesktop-vrdp via this wrapper is just amazing.. audio is just PERFECT!  Absolutely no jitter in the audio at all.  Not only on a local network, but even across WAN.

Its so simple to launch too:  "padsp rdesktop-vrdp -r sound:local .....(whatever other params)"

padsp is part of pulseaudio-utils in Ubuntu

I have no idea why this wrapper works so good.. but kudos to the developers!  Or maybe its just OSS that is so efficient with networked audio??  If anyone knows, let us know in comments below.

Ubuntu LTSP for the Enterprise

Ubuntu LTSP (ltsp-server package) provides a very easy and convenient method for deploying a thin-client environment (see ltsp.org), but is centered primarily around one server being the DHCP/TFTP/NBD and image build server.  Being in an enterprise environment, I needed to be able to split these functions in order to deploy thin clients.  The image below depicts what I needed to achieve. Essentially I wanted to have ubuntu 14.04 images served by older release servers, so my "build" server needs to be 14.04.  Plus I wanted to be able to server multiple custom images off the same server(s).  Note:  I have existing DHCP/TFTP and NBD servers - so this post assumes these are already established - If you do not, then the ltsp-server package has everything you need, and this post is kinda moot.

First, on my build server, I did the following:

apt-get install ltsp-server
ltsp-build-client --dist trusty --arch amd64 --base /opt/ltsp-trusty

This builds your base client system, in my case an amd64 architecture ubuntu 14.04 base.  It takes about 20 minutes depending on your Internet connection and system speed.  Then any customizations can be done via ltsp-chroot:

ltsp-chroot -a amd64 -b /opt/ltsp-trusty --mount-package-cache

(more on what I customized in later post)

And to build the image:

ltsp-update-image --base /opt/ltsp-trusty

So what we have now on the build server, is the image in /opt/ltsp-trusty/images called amd64.img and the needed tftpboot files in /var/lib/tftpboot/ltsp.  In very simple terms, we now need to deploy the image "amd64.img" to the NBD server, and the PXE boot environment to the TFTP server, and then tweak configuration files to point to the right places.

NBD server (Network Block Device)

For every different image you want to deploy, you need nbd-server listening on a different port.  I am using inetd to spawn the nbd-server instances.  Here is my inetd.conf, which is configured for 3 images on ports 2000 thru 2002:


2000  stream  tcp nowait  nobody /usr/sbin/tcpd /usr/sbin/nbdrootd /opt/ltsp/images/i386.img
2001  stream  tcp nowait  nobody /usr/sbin/tcpd /usr/sbin/nbdrootd /opt/ltsp-trusty/images/amd64.img
2002  stream  tcp nowait  nobody /usr/sbin/tcpd /usr/sbin/nbdrootd /opt/ltsp-trusty/images/i386.img

The image files have to be accessible directly, so in my case I copied them from my build server.

TFTP server (Trivial File Transfer Protocol)

For every different image, you need a specific PXE environment defined in /var/lib/tftpboot.  In my case I have 3:

/var/lib/tftpboot/ltsp/i386
/var/lib/tftpboot/ltsp-trusty/amd64
/var/lib/tftpboot/ltsp-trusty/i386

These can be copied as is from the same location on the build server - each environment can have its own lts.conf file.

Now, there is some tweaking required in order for the client to grab the correct NBD image.  You need to add nbdroot={ipaddress}:{port} to the kernel boot parameters for any image served by the non-standard 2000 NBD port.  This is done in the pxelinux.cfg/ltsp file - snippet below.

# This file is regenerated when update-kernels runs.
# Do not edit, see /etc/ltsp/update-kernels.conf instead.
default ltsp-NBD
ontimeout ltsp-NBD

# This file is regenerated when update-kernels runs.
# Do not edit, see /etc/ltsp/update-kernels.conf instead.
label ltsp-NBD
menu label LTSP, using NBD
kernel vmlinuz-3.13.0-37-generic
#append ro initrd=initrd.img-3.13.0-37-generic init=/sbin/init-ltsp quiet splash root=/dev/nbd0
append ro initrd=initrd.img-3.13.0-37-generic init=/sbin/init-ltsp nosplash root=/dev/nbd0 nbdroot=192.168.202.4:2001
ipappend 2

DHCP server (Dynamic Host Configuration Protocol)

Your DHCP configuration is key, as it directs the PXE booting client to the correct TFTP server and PXE path.  I use an Etherboot gPXE image served by a local TFTP server to fix issues with older network cards - but you may not need that.  Here is one client config section example:

host webc2 {     hardware ethernet 20:cf:30:6f:0a:9e;

     fixed-address webc2;

     option host-name "webc2";

     if substring (option vendor-class-identifier, 0, 9) = "PXEClient"           {

   if exists user-class and option user-class = "gPXE" {

      next-server 192.168.202.4;

      filename "/ltsp-trusty/i386/pxelinux.0";

   } else { filename "/tftpboot/undionly.kpxe"; }

} else {

  next-server 192.168.202.4;

  filename "/ltsp-trusty/i386/nbi.img";}}

The "next-server" portion points it to the TFTP server and PXE file.

You'll probably notice that in my case, my TFTP server and NBD server are on the same server, but its not necessary.

Anyway, in another post I'll discuss an improvement I made to LTSP to allow better scripting via TFTP.

Installing Ubuntu 13.04 on an ASUS X55A / X55U

Downloaded the official release of 13.04 yesterday, and tried installing on an ASUS X55A.  I wanted a dual-boot setup.  Took a little bit of effort, but got er done.. steps follow.

First thing I had to do was to disable Fast Boot and disable Secure Boot.  From inital power on, press ESC repeatedly until small blue boot menu appears, then select "Enter Setup".  Goto Boot -> Boot Configuration and disable Fast Boot. Change boot order so that DVD is 1st.  Goto Security and disable Secure Boot Control.  Press F10 to save and exit.  This allows the install media to boot (in my case a DVD).  Install went fine, selected the "Install Ubuntu alongside Windows 8" option.

After install, the system booted to grub, but when selecting Ubuntu, screen would go blank (but not off) and all HDD activity would stop.  Numlock/Capslock keys worked, so system was not hung.  Pressing Enter a few times would cause HDD light to flicker, but otherwise nothing.. Ctrl-Alt-Fn would not do anything.  Strange thing was that if I rebooted, pressed ESC to get blue boot menu, and selected the "ubuntu" option (created by installer), it would goto grub and then boot fine!

To fix this, I installed and ran boot-repair.

sudo add-apt-repository ppa:yannubuntu/boot-repair && sudo apt-get update

 
sudo apt-get install -y boot-repair && (sudo boot-repair &)

I just selected the "recommended" option in boot-repair.  After doing this, I had to add another boot option as per the instructions - had to add sda1/EFI/ubuntu/grubx64.efi.  Pay attention to what it tells you, as it may not be identical, depending on your partitioning.

Reboot and press ESC on boot to get blue menu, Enter Setup, goto Boot -> Add New Boot Option -> Add boot option and give it a name (i used "ubuntu13.04").  The filesystem was pre-selected for me, the path for the boot option I entered as follows:  fsx:sda1/EFI/ubuntu/grubx64.efi  - have no idea what the fsx: is about, but it was given in the help text.  Scroll down to Create, and press enter.  Press ESC to go back, and then move the newly created boot option to #1 position.  Reboot and enjoy.  Fully functional dual-boot system with Win8 and Ubuntu 13.04.

Why the Ubuntu folks don't learn from the boot-repair folks is really a mystery to me...

NOTE:  I ended up having to delete the Windows 8 boot option in the BIOS, because after booting from grub into Win8 once, it did something to bypass grub.. frikin Windows!

Upgrading RT 3.6.4 to RT 4.0.10

Request-Tracker (RT) upgrade steps.  This is based on http://blog.bestpractical.com/2011/07/upgrading-to-rt-4.html.  Please read all related documents and don't just follow this blindly.

Moving from a gentoo linux 2.4 box, RT 3.6.4, mysql 4.0.8 to an ubuntu 12.04 server, RT 4.0.10 and mysql 5.5

Export 3.6.4 Database
On RT 3.6.4 machine as root (obviously must know the root mysql password):

mysqldump -p --opt rt3 > rt3.sql

Copy this file to the new machine.  All further steps are on new machine.

Prepare SQL for Import
You have to edit rt3.sql with vim or sed, and remove all references to "Type=Innodb" (this is default in mysql now), and search for "timestamp(" and remove the bracketed section.. (eg. "timestamp(14)" is now just "timestamp").

Create Database for Import

mysql -p
create database rt4 character set latin1;
grant all on rt4.* to rt_user identified by 'rt_pass';
quit

NOTE: above replace rt_user and rt_pass by the user/password you wish to use for RT.  Also, it may seem counter intuitive to name database rt4, since we are importing rt3 - just remember the end target is 4.0.10.

Import 3.6.4 Database

mysql -p --default-character-set=binary rt4 < rt3.sql

Download/Configure/Install RT 4.0.10

Download RT 4.0.10 source, extract to temp dir, run .configure with needed parameters (see your 3.6.4 install), and basically follow README up to and including "make install".  This should install in /opt/rt4.  Be sure to configure RT_SiteConfig.pm and configure apache2 as needed (beyond scope of this post).

Upgrade Database
From the temp dir used above execute the following:

make upgrade-database
(From 3.6.4 to 3.7.87)

If it fails at 3.7.81, do this:
mysql -p rt4
CREATE INDEX CachedGroupMembers3 on CachedGroupMembers (MemberId, ImmediateParentId);
quit
make upgrade-database
(From 3.7.81 to 3.7.87)

perl etc/upgrade/upgrade-mysql-schema.pl rt4 rt_user rt_pass > queries.sql
(again replacing rt_user and rt_pass with user/password you wish to use for RT)
mysql -p rt4 < queries.sql
(this takes about 3 times as long as the initial import!)
mysql -p
alter database rt4 default character set utf8;
quit
make upgrade-database
(From 3.7.87 to 4.0.9)

If it fails at 3.8.2, do this:
mysql -p rt4
drop index groupmembers1 on GroupMembers;
create unique index groupmembers1 on GroupMembers (GroupId,MemberId);
quit
make upgrade-database
(From 3.8.3 to 4.0.9)

perl -I /opt/rt4/local/lib -I /opt/rt4/lib etc/upgrade/shrink_cgm_table.pl
(takes even longer than above query)
perl -I /opt/rt4/local/lib -I /opt/rt4/lib etc/upgrade/shrink_transactions_table.pl
perl etc/upgrade/vulnerable-passwords --fix

Now test, test, test... :)

Linux memory tweaks for improved responsiveness

Memory management techniques in the linux kernel has changed over the years.  Sometimes a kernel upgrade, or a hardware upgrade, where you expect better performance, can lead to disappointment.

For a while now, I've been struggling with our ubuntu 10.04 application server (managing 35 desktop sessions) where we experience "morning sluggishness" with the system.  Between 6am and noonish, the system was just not as responsive as it could be, but in the afternoons it became quite zippy.  I've been explaining this away with things like "everyone is just logging in morning, system is seeing a heavy load early in the day".. or "everyone does more work in the morning, and take it easy after lunch".

I decided to analyze memory usage patterns combined with I/O wait times.  Between 8am and noon, there was an average I/O wait time of between 2% - 3%, and between noon and 4pm an average of < 1%.  When I graphed out memory usage patterns, I noticed that our nightly backups were consuming a huge amount of buffered memory (understandably).. and that this buffer was slowly reclaimed overtime and around noonish every day it got close to cache size.  This couldn't be coincidence.. performance steadily increased as the buffers from previous nights backup were free'd up.  Makes total sense.. but question was.. how do I modify this behaviour?

Googling around, I came across this link

I already knew about vm.swappiness - already have this set to 10.

But vm.vfs_cache_pressure and vm.drop_caches were new to me.  So I thought, if we can drop the caches right after the backup, and give a little more priority to the inode/dentry caches - this should reduce our I/O wait.

So, I set vm.vfs_cache_pressure to 50 ( sysctl -w vm.vfs_cache_pressure=50 ), and in my backup script, added "echo 3 > /proc/sys/vm/drop_caches" to the end.  The results are quite dramatic.  So far this morning, I have an average I/O wait of  0.19%, down from 2-3%!! (and system is very zippy.)  This graph gives a good visual on the memory usage...

Notice: Everyday at 00:00.. this is backup kicking in.. Tues/Wed shows the slow reclaiming of buffers.  Thurs is where I drop the caches right after backup.. now caches reach previous days peak by 8am!!

Could this be the end of my users complaining about a slow system??   Doubt it... They never notice when its fast.. but always and only notice when its slow.

Update:  After two days now.. amazing.. average daytime I/O wait is now < 0.10%

WinTV 1600 model 1199 and VirtualBox on linux

Had a working VirtualBox WinXP guest on an Ubuntu Natty host.

Installed a WinTV 1600 video capture card, and after that, the VirtualBox WinXP guest would not boot.. got the message:

Failed to load VMMR0.r0 VERR_NO_MEMORY

After digging thru the logs, and noticing vmalloc errors - I decided to blacklist the cx18 modules (for capture card), and reboot. VirtualBox worked now, but then if I removed the blacklist on the cx18 modules, and tried to load the module - I got the same vmalloc errors.

Determined its a shortage of virtual memory in the linux kernel. Default is 128M.

Certain device drivers (modules) - in my case, a video capture card - require chunks of this virtual memory for device mapping.

Virtualbox obviously uses the same mechanism. You can have 16GB of RAM, but if you only have 128M of vm allocated, you are going to run into the same issue.

Solution is to pass the "vmalloc=<size>" parameter to your kernel on boot.  Preferably in 64M chunks.

In ubuntu, edit /etc/default/grub, and add "vmalloc=192M" to the GRUB_CMDLINE_LINUX_DEFAULT list.

eg. GRUB_CMDLINE_LINUX_DEFAULT="vmalloc=192M quiet splash"

sudo update-grub

reboot

Now they play nicely together - and the overall system performance is much snappier.

xscreensaver and empathy presence notification

xscreensaver doesn't support d-bus - and therefore when screensaver activates, empathy (IM client) doesn't change status to "Away", like it does with gnome-screensaver.

The xscreensaver man page includes a nice perl script which you can use to accomplish the same thing.  So here is "watchx.pl" script:

#!/usr/bin/perl

my $blanked = 0;
 open (IN, "xscreensaver-command -watch |");
 while () {
     if (m/^(BLANK|LOCK)/) {
         if (!$blanked) {
             system "/usr/local/bin/empathy-away.sh";
             $blanked = 1;
         }
     } elsif (m/^UNBLANK/) {
         system "/usr/local/bin/empathy-avail.sh";
         $blanked = 0;
     }
 }

Then, using dbus-send, you can set status in empathy.. I use the following scripts "empathy-away.sh":

#!/bin/bash
USER=`whoami`
service=`qdbus | grep $USER | sed -e "s/ //g"`
path=`echo $service | sed -e "s/\./\//g"`
dbus-send --dest=$service /$path org.freedesktop.Telepathy.Connection.Interface.SimplePresence.SetPresence string:"away" string:"Away from keyboard"

and "empathy-avail.sh":

#!/bin/bash
USER=`whoami`
service=`qdbus | grep $USER | sed -e "s/ //g"`
path=`echo $service | sed -e "s/\./\//g"`
dbus-send --dest=$service /$path org.freedesktop.Telepathy.Connection.Interface.SimplePresence.SetPresence string:"available" string:"Available"

NOTE:  This is designed for one empathy account.. and that account name containing the users login name.  Works on ubuntu lucid.  Also - that dbus-send command is all on one line.

And to have it launch at login... create the following "watchx.desktop" file and place it in /etc/xdg/autostart:

[Desktop Entry]
Type=Application
Exec=/usr/local/bin/watchx.pl
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
Name[en_US]=Watch screen saver
Name=Watch screen saver
Comment[en_US]=Watch screen saver
Comment=Watch screen saver

Linux performance tuning - I/O wait

Have a 64-bit ubuntu linux (hardy) system running 30 KDE desktop sessions (using nomachine's NX) on an IBM bladeserver HS21 with 2 quad-core Xeon's and 16G of RAM.  Many times a day, users would experience "freezes" within either firefox or evolution that would last up to about 15 seconds.. window completely unresponsive - then would resume execution.  This using the stock 2.6.24-24-generic ubuntu kernel with voluntary prempting.

After much performance monitoring with sar, and associating the timing of the freezes to sar data, we determined that I/O wait seemed to be the culprit.  Real memory usage {used - (cached+buffered)} only ever peaked around 12G, and CPUs were largely idle.

Disk is 10k rpm 300GB SAS 2.5" drive.. in RAID 1 (mirrored) using HS21 onboard LSI controller.

I/O Wait is primarily disk I/O but could be network I/O as well, so we decided to tune disk and network.

Filesystems is using ext3 and mounted with noatime.

Finding this link , I tried adding the mount options: noatime,nobh,data=writeback,commit=90

NOTE:  If on root filesystem, you must add "rootflags=nobh,data=writeback,commit=90" in grub.

After a reboot, and once full load was back on system, I realized a substantial reduction in average IO wait time (from ~50ms to ~25ms), and overall average % disk utilization (from ~15% to ~8% - statistics from iostat).

This was great improvement but there were still slight 1-2s freezing incidents at this point, so I additionally tweaked the network...

For network performance tuning I ran across this link and added the following to /etc/sysctl.conf:

net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1800
net.core.wmem_max=8388608
net.core.rmem_max=8388608
net.ipv4.tcp_rmem="4096 87380 8388608"
net.ipv4.tcp_wmem="4096 87380 8388608"

Together, these have stopped the "freezing" altogether.  There are still periods of slowness due to load, but now they are few and far between.

Locking Firefox 3 preferences in Ubuntu

Based on http://kirksblog.steffensenfamily.com/archives/24

1. Create a file called loadcustom.js in /usr/lib/firefox-3.0.x/defaults/preferences/, where 3.0.x will be something like 3.0.11 or 3.0.14 depending on your current version.

Put the following in /usr/lib/firefox-3.0.x/defaults/preferences/loadcustom.js

// tell firefox to load customized config file
pref("general.config.obscure_value", 0);
pref("general.config.filename", "firefox.cfg");

2. Create a file called firefox.cfg in /usr/lib/firefox-3.0.x/ with the following content (the first line in both files must start with a comment):

// Lock specific preferences in Firefox so that users cannot edit them
lockPref("app.update.enabled", false);
lockPref("network.proxy.http", "127.0.0.1");
lockPref("network.proxy.http_port", 8080);
lockPref("network.proxy.type", 1);
lockPref("network.proxy.no_proxies_on", "localhost, 127.0.0.1, 192.168.1.0/24");
lockPref("network.proxy.share_proxy_settings", true);
lockPref("browser.startup.homepage", "http://www.desiredhomepage.com/");

3. Restart Firefox, and the preferences should be locked down. You should be able to use this to lock down any setting in about:config

32-bit firefox on 64-bit ubuntu hardy

Problem: External applet which requires Java 1.5 on 64-bit system.. no firefox plugin for Java 1.5 in 64-bit exists to date (Oct. 6, 2009 - and not likely to happen!)

Solution: Run 32-bit firefox in chroot environment

Following https://help.ubuntu.com/community/DebootstrapChroot

sudo apt-get install debootstrap schroot

Edit /etc/schroot/schroot.conf and add the following:

[hardy_i386]

description=Ubuntu 8.04 Hardy for i386

location=/srv/chroot/hardy_i386

personality=linux32

root-users=bob

run-setup-scripts=true

run-exec-scripts=true

type=directory

users=alice,bob,charlie

$ sudo mkdir -p /srv/chroot/hardy_i386

$ sudo debootstrap --variant=buildd --arch i386 hardy /srv/chroot/hardy_i386 http://archive.ubuntu.com/ubuntu/

Will take a while.. downloads base packages

$ schroot -c hardy_i386 -u root

apt-get install ubuntu-minimal

Edit /etc/apt/sources.list and add "universe multiverse" to end of the line

apt-get update

apt-get install firefox sun-java5-plugin

In order to get NIS working, I had to install nis package and edit /etc/nsswitch.conf accordingly.

IF you don't want your "real" /home mounted in chroot environment, then edit /etc/schroot/setup.d/10mount and comment out (#) line with /home.

You'll have to create /home/username (mkdir /home/username; chown username /home/username)

Now exit.. and start again as user:

$ schroot -c hardy_i386

firefox -no-remote

NOTE: -no-remote is need in case 64-bit firefox is running already

Goto "about:plugins" to see if Java plugin is available.

Manually removing ubuntu or debian package

Ran into an issue where a package was only partially removed from an ubuntu system - have no clue why.. but it was in a loop.. "apt-get -f install" would try to remove packagex and fail due to not existant post-remove script.

Thanks to this post you can manually remove an ubuntu/debian package doing the following:

First try:

dpkg -P packagex

Likely this will still fail so do:

dpkg -L packagex

this will give you a list of files to delete when the next step is done.

Then go to the directory:

/var/lib/dpkg/info

and delete the file:

packagex.postrm

and do:

apt-get remove --purge packagex
OR
apt-get -f install

to finish up, remove all files you found in the 'dpkg -L' step, and directories specific to packagex.

Then run apt-get update