520. New node: AMD FX 8350/32 Gb RAM/990 FX

Update 5 Nov 2012: Note that the motherboard doesn't support the CPU and this leads to spontaneous reboots under certain conditions. Make sure to look at the list over supported CPUs for the motherboard you use (in retrospect, obvious -- but as a linux person you get used to ignoring those things since everything's for just OSX or Win).

See here for the troubleshooting thread:
 http://verahill.blogspot.com.au/2013/10/523-random-reboots-troubleshooting-in.html

Also see this thread: http://www.techpowerup.com/forums/showthread.php?t=184061
I'll need to read up on...stuff...but the bottom line seems to be that one would expect issues with this board/cpu combo:

Still only a 4+1 phase board the FX chips pull a bit more power than that can put out comfortably and stable. [..] Those would be your three best to choose from all are the better 8+2 phase designs...

and

my opinion is to stay away from the asus FX ive seen many people asking why their boards are throttling at full load, vrm protection causes voltages to drop at full load when vrms hit a certain temp.

and it seemed that low (CPU) voltages precipitated crashes.

Original post:
So I built a new node at the beginning of October 2013, using the following parts:

• AMD FX 8350 CPU
• 4*8 Gb GSkill RAM
• ASRock 990FX Extreme3 motherboard
• 1 Tb Seagate Barracuda HDD
• MSI N210 graphics card
• ASUS NX1101 Gigabit NIC
• Corsair GS700 PSU
• Antec GX700 case

NOTE that I'm having issues with spontaneous reboots during extended periods (days) of heavy load (100% CPU) which do not appear to be associated with faulty RAM, so you might want to think twice before using the exact same permutation of parts as is listed above. Most likely there's a power issue -- either the PSU isn't supplying enough juice to the Mobo, or the Mobo isn't supplying enough power to the CPU. Note also that the CPU isn't listed as an officially supported CPU by the motherboard manufacturer.*

See here for my troubleshooting thread: http://verahill.blogspot.com.au/2013/10/523-random-reboots-troubleshooting-in.html

So the main value of this post are the photos which show how easy it is to build a computer. Just don't...well...build this particular computer -- use a different motherboard.

The other value of the post is purely personal -- I just wanted to write down the steps to take whenever installing a new node in my little cluster.

There will be a post later on troubleshooting (and hopefully fixing) the issue of the spontaneous reboots.

*I've built a number of computers for myself as well as for other people and haven't had any issues (other than bad RAM) before. I got lazy this time and am paying for it.

---

The first step was assembly:

I like the case -- it's metal and feels robust. Having two fans on top is a definite plus as well as it works well with my home-built rack.

Note that the case doesn't come with a printed manual -- to get the manual you need to go online. And it still falls short -- there's no guide as to how to use the many, many cables it comes with. However, it's not rocket science either. Turns out that the case has a molex plug for powering the case fans. So plug in the molex plug to the PSU, then plug in the fans to the four weird plug/cables that the case comes with. Note that the mobo has no plug for the internal connector USB 3 cable that the case comes with.

See here for more details re the case: http://www.hardocp.com/article/2013/09/12/antec_gx700_atx_computer_case_review

Case, closed

The glorious innards of The Case.

The 700GX does not come with a mobo panel -- not that they tend to be useful anyway

Luckily all (most?) mobos come with their own panels -- push it in place before doing anything else. It can need a bit of negotiation in order to snap in properly.

The case came with riser nuts, four PSU screws and lots of screws for the mobo.

Put the riser nuts in the case -- they are the golden thingies

And here's the mother board. I don't know if there are universally accepted recommendations, but I prefer to install the CPU and RAM onto the mother board before installing it in the case -- you have more space to manouver and the risk of breaking the board is smaller.

The heatsink (left) and CPU (right)

The heatsink comes with thermal paste pre-applied. Don't touch it -- you want it to be as smooth and even as possible.

Get the CPU out

Note the yellow triangle in the bottom right corner in the picture

That should match up with the triangle in the bottom left of this picture. Note the raised level on the right side of the CPU socket.

The CPU in place. Note the raised lever. There should be no pushing -- the CPU should fit perfectly without any force whatsoever. If you bend a pin...then good luck.

The lever is in the locked position.

Next put the heatsink on. Give this a bit of thought as you won't want to have to reseat it several times (in the worst case you'll have to go buy some thermal paste, clean the heatsink and reapply the paste). So make sure you line up the fasteners before pushing the heatsink in place.

Everything is locked down.

The motherboard with the processor and heatsink in place. In the picture the CPU fan is attached to the WRONG connector. Look for a connector saying 'CPU FAN' (in the picture it's attached to POWER FAN)

Open the RAM slot fasteners, and push the RAM sticks in place firmly, but without excessive violence. Once the fasteners snap shut by themselves the sticks are properly seated. Improperly seated RAM sticks tend to prevent you from booting and leads to a lot of noise.

All four RAM sticks in place, and the motherboard attached to the case via seven screws that screw into the riser nuts.

The PSU is in place.

Main power and auxiliary power cables attached.

This particular case has a special tray for the hard drives.

Hard drive in place

SATA data and power cables attached

The other end of the SATA data cable attaches to the motherboard (SATA 1)

After a bit of rewiring. 

PCI NIC and PCI-E graphics cards in place.

And below is a picture of the cluster -- each node is connected to a gigabit WAN (192.168.2.0/24) router  and a gigabit LAN switch (192.168.1.0/24). 8/32 means 8 cores, 32 gb ram. The cluster 'runs' on the LAN. Each of the four nodes in the picture (there are two three-core nodes in addition) are connected to a KVM. Jobs are managed using SGE.

It's questionable whether one can really call it a cluster though since I run each job on a single node for performance reasons. It still attracts attention from visitors to my office though.

---

Software:
I then installed debian wheezy on it. During the installation I was notified that I might want to consider enabling non-free to get the r8169 and tg3 firmwares

So after enabling non-free in the sources I did:
sudo apt-get install firmware-realtek firmware firmware-linux-nonfree

Didn't seem to change anything though -- everything was working fine before too.

I also installed amd64-microcode which, if I understand things correctly, should obviate the need for some of the full BIOS updates.

Other little housekeeping things:
I first sorted out
INIT: Id "co" respawning too fast: disabled for 5 minutes
as shown here: http://verahill.blogspot.com.au/2012/01/debian-testing-64-wheezy-small-fixes.html

I then installed a few basic thing:
sudo apt-get install vim screen sinfo gawk lm-sensors

and made a ~/.vimrc:

set number
set pastetoggle=<f3>
nnoremap <f4> :set nonumber!<CR>

And set vim to the default editor in lieu of nano:

sudo update-alternatives --config editor

I edited /etc/default/sinfo to make it use the correct network:

OPTS="${OPTS} --quiet --bcastaddress=192.168.1.255"

I set up 'static' dhcp on the WAN router.

On the node, I then sorted out /etc/network/interfaces  to use dhcp on eth1 and 192.168.1.180 on eth0, and to route everything properly (i.e. local traffic over eth0, and everything else over eth1):

auto lo
iface lo inet loopback

auto eth1
iface eth1 inet dhcp

auto eth0
iface eth0 inet static
address 192.168.1.180
gateway 192.168.1.1
netmask 255.255.255.0

post-up ip route flush all
post-up route add default eth1
post-up route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.1 eth0

SGE won't work properly unless you edit /etc/hosts:

127.0.0.1       localhost
#127.0.1.1      oxygen
192.168.1.180   oxygen

The way my cluster works is that every node has its own shared folder.

mkdir ~/oxygen
mkdir ~/scratch
chmod 777 ~/oxygen

Export it as shown here: http://verahill.blogspot.com.au/2012/02/debian-testing-wheezy-64-sharing-folder.html
Set up ssh key login in both directions:

ssh-keygen
vim ~/.ssh/authorized_keys

Then add the new node to the cluster: http://verahill.blogspot.com.au/2013/08/501-briefly-adding-new-node-to-sge.html
Build nwchem as shown here: http://verahill.blogspot.com.au/2013/05/424-nwchem-63-on-debian-wheezy.html
Set up gaussian as shown here: http://verahill.blogspot.com.au/2012/05/settiing-up-gaussian-g09-on-debian.html
Fix shmem: http://verahill.blogspot.com.au/2012/10/shmmax-revisited-and-shmall-shmmni.html

Finally, to address this issue regarding corrupt packages during SSH sessions I then added to /etc/rc.local: /sbin/ethtool -K eth1 rx off tx off

519. Formatting an XYZ molecular geometry file using python

This is a silly little script -- ECCE, which I use to manage all my computations, is very particular about what XYZ files it can and cannot read -- if the number of spaces between the coordinates are wrong, it will fail to read the structure. So here's a python script, based on parts I've cannibalised from other scripts that I've written in the past (i.e. it could be made more elegant, but I had the parts ready and just wanted something that works) which takes a somewhat ugly XYZ file and turns it into something beautiful. At least in the eyes of ECCE.

The impetus for this was that someone gave me an XYZ file the had generated by copying columns from Excel. On Mac. mac2unix took care of the line endings, but there were tabs (\t) all over the place.

Usage:

polish_xyz ugly.xyz pretty.xyz

Script:

#!/usr/bin/python
import sys 

def getrawdata(infile):
    f=open(infile,'r')
    n=0 
    preamble=[]
    struct=[]
    
    for line in f:
        if n<2: data-blogger-escaped-if="" data-blogger-escaped-line.rstrip="" data-blogger-escaped-n="" data-blogger-escaped-preamble="">3:
            line=line.rstrip()
            struct+=[line]
        n+=1
    xyz=[struct]
    return xyz, preamble

def genxyzstring(coords,elementnumber):
    x_str='%10.5f'% coords[0]
    y_str='%10.5f'% coords[1]
    z_str='%10.5f'% coords[2]
    element=elementnumber
    xyz_string=element+(3-len(element))*' '+10*' '+\ 
    (8-len(x_str))*' '+x_str+10*' '+(8-len(y_str))*' '+y_str+10*' '+(8-len(z_str))*' '+z_str+'\n'
    
    return xyz_string

def getstructures(rawdata,preamble,outfile):
    n=0 
    for structure in rawdata:
        n=n+1
        num="%03d" % (n,)
        g=open(outfile,'w')
        itson=False
        cartesian=[]
    
        for item in structure:
            coordx=filter(None,item.split(' ')) 
            coordy=filter(None,item.split('\t'))
            if len(coordx)>len(coordy):
                coords=coordx
            else:
                coords=coordy
            coordinates=[float(coords[1]),float(coords[2]),float(coords[3])]
            element=coords[0]
            cartesian+=[genxyzstring(coordinates,element)]
        g.write(str(preamble[0])+'\n')
        g.write(str(preamble[1])+'\n')
        for line in cartesian:
            g.write(line)
        g.close()
        cartesian=[]
    return 0

if __name__ == "__main__":
    infile=sys.argv[1]
    outfile=sys.argv[2]
    xyz,preamble=getrawdata(infile)
    structures=getstructures(xyz,preamble,outfile)

518. Generating enantiomers of molecular structures given in XYZ coordinates using python.

What I'm showing here is probably overkill -- there may be better ways of doing this with sed/awk*. However, since I had most of the code below ready as it was part of another script, doing this is python was quick and easy. Plus it's portable-ish.

*[ECCE, which I use for managing my calculations, is very, very particular about the format of the XYZ file, including the number of chars between coordinates. So simply doing e.g.

cat molecule.xyz|gawk '{print $1,$2,$3,-$4}'

won't work. Not all pieces of software are that picky when it comes to xyz coordinates though -- jmol, for example, is very forgiving.]

Anyway, the script below flips a molecular structure by taking the geometry given as a (properly formatted) XYZ file, and changing the sign in front of the Z coordinates. It's that simple.
Save it, call it flip_xyz, make it executable and call it using e.g.

flip_xyz molecule.xyz flipped_molecule.xyz

Script:

#!/usr/bin/python
import sys 

def getrawdata(infile):
    f=open(infile,'r')
    n=0 
    preamble=[]
    struct=[]
  
    for line in f:
        if n<2: data-blogger-escaped-if="" data-blogger-escaped-line.rstrip="" data-blogger-escaped-n="" data-blogger-escaped-preamble="">1:
            line=line.rstrip()
            struct+=[line]
        n+=1
    xyz=[struct]
    
    return xyz, preamble

def genxyzstring(coords,elementnumber):
    x_str='%10.5f'% coords[0]
    y_str='%10.5f'% coords[1]
    z_str='%10.5f'% -coords[2]
    element=elementnumber
    xyz_string=element+(3-len(element))*' '+10*' '+\ 
    (8-len(x_str))*' '+x_str+10*' '+(8-len(y_str))*' '+y_str+10*' '+(8-len(z_str))*' '+z_str+'\n'
    
    return xyz_string

def getstructures(rawdata,preamble,outfile):
    n=0 
    for structure in rawdata:
        n=n+1
        num="%03d" % (n,)
        g=open(outfile,'w')
        cartesian=[]
    
        for item in structure:
            coordx=filter(None,item.split(' ')) 
            coordy=filter(None,item.split('\t'))
            if len(coordx)>len(coordy):
                coords=coordx
            else:
                coords=coordy
    
            coordinates=[float(coords[1]),float(coords[2]),float(coords[3])]
            element=coords[0]
            cartesian+=[genxyzstring(coordinates,element)]
    
        g.write(str(preamble[0])+'\n')
        g.write(str(preamble[1])+'\n')
        for line in cartesian:
            g.write(line)
        g.close()
        cartesian=[]
    return 0

if __name__ == "__main__":
    infile=sys.argv[1]
    outfile=sys.argv[2]
    xyz,preamble=getrawdata(infile)
    structures=getstructures(xyz,preamble,outfile)

517. Very briefly: Prime95 (GIMPS) on linux

I'm very unhappy about a newly built node which randomly crashes and reboots when running long jobs. More about that later, but here are the specs: FX 8350, 4x8 Gb RAM GSkill Ripjaws, ASRock FX990 Extreme3, Corsair GS700, MSI N210, ASUS NX1101 in an Antec GX700 case, running Wheezy with stock kernel (3.2.0-4 amd64).

I've tested the RAM using memtest86+  and found no errors, the rig uses a 700 W Corsair PSU which /should/ provide enough power, and I see no evidence of overheating based on a cronjob which runs every 2 minutes. Anyway, the first step in troubleshooting is finding a good way of reproducing the error reliably, and prime95 is what the windows overclockers use to stresstest.

Turns out prime95 (actually GIMPS) can run in a few different modes which tests different aspects of you system, which makes it sound like a pretty good program for my purposes.

See here for more information: http://www.mersenne.org/freesoft/

---

mkdir ~/tmp/mprime -p
cd ~/tmp/mprime
wget http://www.mersenne.info/gimps/p95v279.linux64.tar.gz
tar xvf p95v279.linux64.tar.gz
./mprime

Welcome to GIMPS, the hunt for huge prime numbers.  You will be asked a 
few simple questions and then the program will contact the primenet server 
to get some work for your computer.  Good luck!

Attention OVERCLOCKERS!!  Mprime has gained a reputation as a useful 
stress testing tool for people that enjoy pushing their hardware to the 
limit.  You are more than welcome to use this software for that purpose.  
Please select the stress testing choice below to avoid interfering with 
the PrimeNet server.  Use the Options/Torture Test menu choice for your 
stress tests.  Also, read the stress.txt file.

If you want to both join GIMPS and run stress tests, then Join GIMPS and 
answer the questions.  After the server gets some work for you, stop 
mprime, then run mprime -m and choose Options/Torture Test.

Join Gimps? (Y=Yes, N=Just stress testing) (Y): N
Number of torture test threads to run (3): 2
Choose a type of torture test to run.
1 = Small FFTs (maximum FPU stress, data fits in L2 cache, RAM not tested 
much).
2 = In-place large FFTs (maximum heat and power consumption, some RAM 
tested).
3 = Blend (tests some of everything, lots of RAM tested).
11,12,13 = Allows you to fine tune the above three selections.
Blend is the default.  NOTE: if you fail the blend test, but can pass the 
small FFT test then your problem is likely bad memory or a bad memory 
controller.
Type of torture test to run (3): 1
Accept the answers above? (Y): Y
[Main thread Sep 20 11:06] Starting workers.
[Worker #1 Sep 20 11:06] Worker starting
[Worker #1 Sep 20 11:06] Setting affinity to run worker on any logical CPU.
[Worker #2 Sep 20 11:06] Worker starting
[Worker #2 Sep 20 11:06] Setting affinity to run worker on any logical CPU.
[Worker #1 Sep 20 11:06] Beginning a continuous self-test to check your computer.
[Worker #1 Sep 20 11:06] Please read stress.txt.  Hit ^C to end this test.
[Worker #2 Sep 20 11:06] Beginning a continuous self-test to check your computer.
[Worker #2 Sep 20 11:06] Please read stress.txt.  Hit ^C to end this test.
[Worker #1 Sep 20 11:06] Test 1, 180000 Lucas-Lehmer iterations of M580673 using AMD K10 type-1 FFT length 28K, Pass1=112, Pass2=256.
[Worker #2 Sep 20 11:06] Test 1, 180000 Lucas-Lehmer iterations of M580673 using AMD K10 type-1 FFT length 28K, Pass1=112, Pass2=256.
CTRL+C

And so on.

516. Potential energy diagrammes in latex using pgfplots

Given these recent posts, a post on how to represent reaction coordinates in latex was inevitable.

Note: a much prettier example is found here: https://quantumchemistryniser.wordpress.com/2013/05/25/potential-energy-profile-using-pgfplots/

However, I initially couldn't reproduce it, and after spending a few hours with pgfplots I wanted my own style anyway.

---

But first, a recap:
Once you've completely left office-like wordprocessors behinds and fully embraced the wonder of typesetting in LaTeX you find yourself wanting to produce more and more of the figures directly in latex, rather than simply importing figures.

While nothing beats gnuplot for plotting, and pov-ray (via e.g. gdis) for fancy figures and bkchem for simple 2d structures, there are some nice and simple latex packages that make life easier: I touch on a few in this post: http://verahill.blogspot.com.au/2012/07/debian-texlive-and-style-files-making.html

For chemical formulae, use mhchem -- I think it's standard in debian for texlive. It's indispensable. Also see http://www.ctan.org/pkg/mhchem

For very, very simple chemical structures you can look at chemfig -- I don't use it much though. http://www.ctan.org/tex-archive/macros/latex/contrib/chemfig/

For drawing simple MO diagrammes, modiagram is very pretty: http://www.ctan.org/pkg/modiagram

For drawing NMR pulse sequences there are a few options: http://verahill.blogspot.com.au/2013/08/499-briefly-drawing-nmr-sequences-using.html and http://verahill.blogspot.com.au/2013/08/498-briefly-drawing-nmr-pulse-sequences.html

---

A dead end: endiagram
It looks very promising, but it didn't work out for me: http://ctan.mirrorcatalogs.com/macros/latex/contrib/endiagram/

Installation:

mkdir ~/texmf/tex/latex/endiagram -p
cd ~/texmf/tex/latex/endiagram
wget http://ctan.mirrorcatalogs.com/macros/latex/contrib/endiagram/endiagram.sty
sudo texhash

You'll need siunitx and tikz as well, but it seems to come with a standard texlive install on debian. Out of the box I kept getting:

! Undefined control sequence.

<argument> \clist_count:N

\l__endiagram_points_clist

l.16 \ENcurve{1,4,0}

In the end I couldn't get it to work properly, even after installing l3kernel -- I don't think it would've been impossible to figure it out, but I was running out of patience.

---

The solution: pgfplots
So I kept looking, and found this: https://quantumchemistryniser.wordpress.com/2013/05/25/potential-energy-profile-using-pgfplots/

While I couldn't get it to work 'out of the box' (after deleting the nodes depending on the pdf files), I ended up reading the pfgplots manual: http://pgfplots.sourceforge.net/pgfplots.pdf

pgfplots is not meant for drawing potential energy diagrammes, but it can be made to do it -- it's 'dum' in the sense that it's not aware of what a PE diagram is, so you'll have to draw all objects expliticly. In the end, it's actually quite simple though.

I eventually ended up with these two examples:

The code is here:

\documentclass[10pt,a4paper]{article}
\usepackage{pgfplots}
\usepackage{tikz}
\begin{document}

%\begin{center}
%\begin{tikzpicture}[scale=1.0]
%\draw (0,0) --(1,2);
%\end{tikzpicture}
%\end{center}

\begin{tikzpicture}
\begin{axis}[
 axis x line= bottom,
 axis y line= left,
 xmin=-1,   xmax=10,
 ymin=-1,   ymax=6,
 xlabel=Reaction coordinate,
 ylabel=$\Delta$E (kcal/mol),
 xtick=\empty
]
 \addplot[smooth,solid,blue] coordinates { (0,0)(1,0) (2,4)(3,4) (4,3)(5,3) (6,4)(7,4) (8,1)(9,1)};
 \node at (axis cs:4.5,2.5) {Intermediate};
\end{axis}
\end{tikzpicture}

\begin{tikzpicture}
\begin{axis}[
 axis x line= bottom,
 axis y line= left,
 xmin=-1,   xmax=6,
 ymin=-1,   ymax=15,
 xlabel=Reaction coordinate,
 ylabel=$\Delta$E (kcal/mol),
 xtick=\empty
]
 \addplot[smooth,solid,blue] coordinates { (0,0)(1,0)};
 \addplot[smooth,dotted,black] coordinates {(1,0)(2,11.7)};
 \addplot[smooth,solid,blue] coordinates {(2,11.7)(3,11.7)}; 
 \addplot[smooth,dotted,black] coordinates {(3,11.7)(4,10.5)};
 \addplot[smooth,solid,blue] coordinates {(4,10.5)(5,10.5)};

\end{axis}
\end{tikzpicture}
\end{document}

515. Very briefly: boot issues and nvram (anecdote)

I had to move offices at work last Friday (which partly explains my radio silence).

One of my nodes, with a Biostar 880G+ motherboard, wouldn't boot. The issue manifested itself in a few different ways (and not always at the same time):

Most of the time the bios portion of the boot would be very slow, and I'd get a loooooong beep when the voltages were echoed during what I presume is POST. I never made it to the GRUB screen. I'd had this happen once or twice before, especially when the computer had been turned off and/or unplugged for a longer period of time.

Occasionally I'd get a message about NVRAM being checked..

After
1) having checked that nothing had vibrated loose during the transport
2) having removed all cards and drives except a single RAM stick
3) having checked that RAM stick
I still wasn't getting any closer to getting booted. So I went into the BIOS. Having seen the message about nvram and having done a bit of reading online I selected to erase the NVRAM.

And suddenly booting worked...once.

After that I kept getting messages about nvram being corrupted and boot fail, or 'NVRAM...Update failed'.

I then set the CMOS jumpers to pin 2 and 3 (small jumper next to the CMOS battery)

And then it worked! It booted just fine! But I kept getting the following error echoing on my screen:

[  120.718683] hub 3-0:1.0: unable to enumerate USB device on port 2
[  120.932378] usb 3-2: new full-speed USB device number 28 using ohci_hcd
[  121.072497] usb 3-2: device descriptor read/64, error -62
[  121.316673] usb 3-2: device descriptor read/64, error -62
[  121.556904] usb 3-2: new full-speed USB device number 29 using ohci_hcd
[  121.697019] usb 3-2: device descriptor read/64, error -62
[  121.941224] usb 3-2: device descriptor read/64, error -62
[  122.181425] usb 3-2: new full-speed USB device number 30 using ohci_hcd

Unplugged all the USB devices. Turned out to be the KVM (ritmo). Turning it off (i.e. unplugging all USB input to the KVM so that it lost power) and plugging it in again solved it.

And ever since then all has been fine.

Presumably it's time to change the CMOS battery.

514. Extracting Frequency data from a gaussian 09 calculation for gnuplot

This is another python script.

Say you've done a computation along the lines of this:

#P rBP86/GEN 5D Pseudo(Read) Opt=() Freq=() SCF=(MaxCycle=999 ) Punch=(MO) Pop=()

and want the data in a neat data file, like this:

33.237 0.0023 0.0536
39.9976 0.0043 0.8305
69.7345 0.0129 0.3348
84.7005 0.0173 0.7027
[..]
3133.0068 6.2938 0.6114
3143.8021 6.3551 0.3775
3164.9242 6.4685 0.8829
3221.8787 6.6972 4.6005

Then you can use the following python (2.x) script, g09freq:

#!/usr/bin/python
# Compatible with python 2.7 
# Reads frequency output from a g09 (gaussian) calculation
# Usage ex.: g09freq g09.log ir.dat
import sys 

def ints2float(integerlist):
    for n in range(0,len(integerlist)):
        integerlist[n]=float(integerlist[n])
    return integerlist

def parse_in(infile):
    g09output=open(infile,'r')
    captured=[]
    for line in g09output:
        if ('Frequencies' in line) or ('Frc consts' in line) or ('IR Inten' in line):
            captured+=[line.strip('\n')]
    g09output.close()
    return captured
    
def format_captured(captured):
    vibmatrix=[]
    steps=len(captured)
    for n in range(0,steps,3):
        freqs=ints2float(filter(None,captured[n].split(' '))[2:5])
        forces=ints2float(filter(None,captured[n+1].split(' '))[3:6])
        intensities=ints2float(filter(None,captured[n+2].split(' '))[3:6])
        for m in range(0,3):
            vibmatrix+=[[freqs[m],forces[m],intensities[m]]]
    return vibmatrix

def write_matrix(vibmatrix,outfile):
    f=open(outfile,'w')
    for n in range(0,len(vibmatrix)):
        item=vibmatrix[n]
        f.write(str(item[0])+'\t'+str(item[1])+'\t'+str(item[2])+'\n')
    f.close()
    return 0

if __name__ == "__main__":
    infile=sys.argv[1]
    outfile=sys.argv[2]

    captured=parse_in(infile)

    if len(captured)%3==0:
        vibmatrix=format_captured(captured)
    else:
        print 'Number of elements not divisible by 3 (freq+force+intens=3)'
        exit()
    success=write_matrix(vibmatrix,outfile)
    if success==0:
        print 'Read %s, parsed it, and wrote %s'%(infile,outfile)

Run it as e.g.

g09freq g09.log test.out

The output is compatible with gnuplot:

gnuplot> set xrange [3500:0]
gnuplot> set yrange [10:-1]
gnuplot> plot './test.out' u 1:2 w impulse

It's trivial to add gaussian broadening (see e.g. this post)

513. Extracting data from a PES scan with gaussian

There are a few reasons to like gaussian, and many reasons not to. Gaussian is fast, and their whitepapers are great resources for learning computational techniques.

Without going into discussions about the commercial behaviour of Wavefunction inc., the things I don't like about gaussian is the clunky input format (nwchem has a much more readable syntax), the inscrutable error messages, and the unreadable output. Well, it's not unreadable in a literal sense, but it could certainly be clearer. On the other hand, I've having issues with running some of my PES scans in nwchem -- and I can't find a solution (more about that in a later post)

Anyway, here's a python script for extracting optimized structures and energies from a relaxed PES scan in Gaussian 09.

First, an example of a simple scan:

%nprocshared=2
%Chk=methanol.chk
#P rB3LYP/6-31g 6D 10F Opt=(modredundant) NoSymm  Punch=(MO) Pop=() 

methanol

0 1 ! charge and multiplicity
 C     0.0351714     0.00548884     0.0351714
 H     -0.617781     -0.634073     0.667983
 H     0.667983     -0.634073     -0.617781
 H     -0.605139     0.646470     -0.605139
 O     0.839603     0.818768     0.839603
 H     1.38912     0.201564     1.38912

 1 5 S 10 0.1

And here's the script, pes_parse_g09:

#!/usr/bin/python
import sys

def getrawdata(infile):
        f=open(infile,'r')
        opt=0
        geo=0
        struct=[]
        structure=[]
        energies=[]
        energy=[]
        for line in f:
                
                if opt==1 and geo==1 and not ("---" in line):
                        structure+=[line.rstrip()]
                
                if 'Coordinates (Angstroms)' in line:
                        if opt==0:
                                opt=1
                                structure=[]
                        
                if opt==1 and "--------------------------" in line:
                        if geo==0:
                                geo=1
                        elif geo==1:
                                geo=0
                                opt=0
                if 'SCF Done' in line:
                        energy=filter(None,line.rstrip('\n').split(' '))
                if      'Optimization completed' in line and (opt==0 and geo==0):
                        energies+=[float(energy[4])]
                        opt=0
                        geo=0
                        struct+=[structure]
                        structure=[]
        
        return struct, energies

def periodictable(elementnumber):
        ptable={1:'H',2:'He',\
        3:'Li', 4:'Be',5:'B',6:'C',7:'N',8:'O',9:'F',10:'Ne',\
        11:'Na',12:'Mg',13:'Al',14:'Si',15:'P',16:'S',17:'Cl',18:'Ar',\
        19:'K',20:'Ca',\
        21:'Sc',22:'Ti',23:'V',24:'Cr',25:'Mn',26:'Fe',27:'Co',28:'Ni',29:'Cu',30:'Zn',\
        31:'Ga',32:'Ge',33:'As',34:'Se',35:'Br',36:'Kr',\
        37:'Rb',38:'Sr',\
        39:'Y',40:'Zr',41:'Nb',42:'Mo',43:'Tc',44:'Ru',45:'Rh',46:'Pd',47:'Ag',48:'Cd',\
        49:'In',50:'Sn',51:'Sb',52:'Te',53:'I',54:'Xe',\
        55:'Cs',56:'Ba',\
        57:'La',58:'Ce',59:'Pr',60:'Nd',61:'Pm',62:'Sm',63:'Eu',64:'Gd',65:'Tb',66:'Dy',67:'Ho',68:'Er',69:'Tm',70:'Yb',71:'Lu',\
        72:'Hf', 73:'Ta', 74:'W',75:'Re', 76:'Os', 77:'Ir',78:'Pt', 79:'Au', 80:'Hg',\
        81:'Tl', 82:'Pb', 83:'Bi',84:'Po',85:'At',86:'Rn',\
        87:'Fr',88:'Ra',\
        89:'Ac',90:'Th',91:'Pa',92:'U',93:'Np',94:'Pu',95:'Am',96:'Cm',97:'Bk',98:'Cf',99:'Es',100:'Fm',101:'Md',102:'No',\
        103:'Lr',104:'Rf',105:'Db',106:'Sg',107:'Bh',108:'Hs',109:'Mt',110:'Ds',111:'Rg',112:'Cn',\
        113:'Uut',114:'Fl',115:'Uup',116:'Lv',117:'Uus',118:'Uuo'}
        element=ptable[elementnumber]
        return element

def genxyzstring(coords,elementnumber):
        x_str='%10.5f'% coords[0]
        y_str='%10.5f'% coords[1]
        z_str='%10.5f'% coords[2]
        element=periodictable(int(elementnumber))
        xyz_string=element+(3-len(element))*' '+10*' '+\
        (8-len(x_str))*' '+x_str+10*' '+(8-len(y_str))*' '+y_str+10*' '+(8-len(z_str))*' '+z_str+'\n'
 
        return xyz_string

def getstructures(rawdata):
        
        n=0
        for structure in rawdata:
                
                n=n+1
                num="%03d" % (n,)
                g=open('structure_'+num+'.xyz','w')
                itson=False
                cartesian=[]
                        
                for item in structure:
                        
                        coords=filter(None,item.split(' '))
                        coordinates=[float(coords[3]),float(coords[4]),float(coords[5])]
                        element=coords[1]
                        cartesian+=[genxyzstring(coordinates,element)]
                g.write(str(len(cartesian))+'\n')
                g.write('Structure '+str(n)+'\n')
                for line in cartesian:
                        g.write(line)
                g.close()
                cartesian=[]
        return 0
        
if __name__ == "__main__":
        infile=sys.argv[1]
        rawdata,energies=getrawdata(infile)
        structures=getstructures(rawdata)
        g=open('energies.dat','w')
        for n in range(0,len(energies)):
                g.write(str(n)+'\t'+str(energies[n])+'\n')
        g.close()

And here's what we get from the output:

g09 methanol.in |tee methanol.out
pes_parse_g09 methanol.log
cat structure* > meoh_traj.xyz

And here's a plot of energies.dat:

512. Briefly: zmatrices in nwchem -- methanol

And another update:
I can now confirm that using your own z matrix still does not constrain the geometry during a PES scan, which was the original impetus for this post: http://verahill.blogspot.com.au/2013/09/511-when-nwchem-pes-scans-fail-to.html

Another update:
the gaussian run failed after 14 geometry steps during the first PES point.

NTrRot=    -1 NTRed=   628 NAtoms=    34 NSkip=   532 IsLin=F
 Error in internal coordinate system.
 Error termination via Lnk1e in /opt/gaussian/g09/l103.exe at Thu Sep  5 18:17:12 2013. 
 Job cpu time:  0 days 22 hours 25 minutes 27.6 seconds.
 File lengths (MBytes):  RWF=    192 Int=      0 D2E=      0 Chk=     28 Scr=      1 

Not being an expert, to me it seems that there's something fundamentally difficult with the system I'm working on. In an ideal world I'd give the actual details, but quite apart from the risk of being scooped, doing so would also make it easier to identify me (not that it's impossible at this point).

[Suffice to say that the system holds a large polyoxoanion and a small p-block anion, both of which are symmetrical and negatively charged. The goal of the PES scan is to bring the ions closer to see whether they 'react'. Which is also a troublesome use of computational resources -- computational chemistry is good at answering well-defined questions using carefully designed computational experiments -- but not generally very good at answering ill-defined questions about synthesis (i.e. you can't generally 'mix two things together and see what happens' and expect a useful result. Anyway, regardless of that, that's exactly what I want to do.]

Update:
nwchem still gives errors about autoz in spite of using noautoz. But I also get messages about the user generated z matrix, so we'll see whether my input is respected or not.

Also, for one of the calcs I'm getting

There are insufficient internal variables: expected   95 got   96

which is really, really, really annoying since there doesn't seem to be a real fix for it -- I've tried everything suggested in http://www.nwchem-sw.org/index.php/Special:AWCforum/st/id286. I can get the same calc to run in gaussian though (gaussian has its own issues), but it'd be nice if stuff just...worked...

Original post:
Normally you don't have to fiddle with zmatrices in nwchem -- instead you'd typically supply cartesian coordinates, and nwchem would do autoz to autogenerate internal (z matrix) coordinates.

Sometimes that fails, and nwchem defaults to using cartesian coordinates. In most cases, this isn't a cause for any real concern -- the computation will continue although I think cartesian coordinates are supposed to be slightly slower.

However, if you're doing a PES scan you'll notice that it's not proceeding as intended -- the constraints are completely ignored: 511. When nwchem PES scans fail to constrain -- autoz failure

The easiest remedy is to supply the internal coordinates directly, but there honestly aren't too many examples online showing how that's done, and I kept on getting annoying failure messages along the lines of

NWChem Input Module
                                -------------------


                                       zmat
                                       ----
 THE 3-D  PIECE OF -Z- DATA FOR ATOM =    2 IS NEITHER FLOATING POINT
 NOR ALPHANUMERIC OR COULD NOT BE MATCHED WITH A VARIABLE. STOP
 IAT=    2 ZMAT=  2  1  0  0  0     0.00000     0.00000     0.00000

 ------------------------------------------------------------------------
                               JOB STOPPED

 PROGRAM STOP IN - ZDAT -

 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  CALLS IT QUIT FROM HND_HNDERR         0
 ------------------------------------------------------------------------
 This error has not yet been assigned to a category

This particular error came about because the zmatrix module is case sensitive, and my Variables couldn't be interpreted (it should be variables). Anyway, you'll understand more after this post, and it isn't important anyway.

---

Calculation using a z matrix (internal coordinates) in nwchem, with a little bit of help from openbabel:

Assuming that you set up a calculation in e.g. ECCE for a geometry optimisation of methanol you'll end up with the following input file:

scratch_dir /home/andy/scratch
Title "methanol"

Start  methanol

echo

charge 0

geometry autosym units angstrom
 C     0.0351714     0.00548884     0.0351714
 H     -0.617781     -0.634073     0.667983
 H     0.667983     -0.634073     -0.617781
 H     -0.605139     0.646470     -0.605139
 O     0.839603     0.818768     0.839603
 H     1.38912     0.201564     1.38912
end

ecce_print ecce.out

basis "ao basis" spherical print
  H library "6-31+G*"
  O library "6-31+G*"
  C library "6-31+G*"
END

dft
  mult 1
  direct
  XC b3lyp
  grid fine
  mulliken
end

driver
  default
end

task dft optimize

Take the coordinates, and paste them into a file, e.g. methanol.xyz:

6
methanol
 C     0.0351714     0.00548884     0.0351714
 H     -0.617781     -0.634073     0.667983
 H     0.667983     -0.634073     -0.617781
 H     -0.605139     0.646470     -0.605139
 O     0.839603     0.818768     0.839603
 H     1.38912     0.201564     1.38912

Next, use openbabel:

babel -ixyz methanol.xyz -ogzmat 

#Put Keywords Here, check Charge and Multiplicity.

 methanol

0  1
C
H  1  r2
H  1  r3  2  a3
H  1  r4  2  a4  3  d4
O  1  r5  2  a5  3  d5
H  5  r6  1  a6  2  d6
Variables:
r2= 1.1117
r3= 1.1117
a3= 109.74
r4= 1.1094
a4= 108.78
d4= 118.90
r5= 1.3984
a5= 110.18
d5= 238.51
r6= 0.9924
a6= 105.98
d6=  60.61

1 molecule converted
18 audit log messages

The format isn't quite right (everything in red needs to go, and the V in blue should be lower case), but we can sort that out:

 babel -ixyz ~/methanol.xyz -ogzmat |sed 's/\=//g;s/V/v/g;s/\://g' |tail -n+6 > methanol.zmat

C
H  1  r2
H  1  r3  2  a3
H  1  r4  2  a4  3  d4
O  1  r5  2  a5  3  d5
H  5  r6  1  a6  2  d6
variables
r2 1.1117
r3 1.1117
a3 109.74
r4 1.1094
a4 108.78
d4 118.90
r5 1.3984
a5 110.18
d5 238.51
r6 0.9924
a6 105.98
d6  60.61

Let's update out nwchem input file with the internal coordinates:

scratch_dir /home/andy/scratch
Title "methanol"

Start  methanol

echo

charge 0

geometry noautoz
zmatrix
C
H  1  r2
H  1  r3  2  a3
H  1  r4  2  a4  3  d4
O  1  r5  2  a5  3  d5
H  5  r6  1  a6  2  d6
variables
r2 1.1117
r3 1.1117
a3 109.74
r4 1.1094
a4 108.78
d4 118.90
r5 1.3984
a5 110.18
d5 238.51
r6 0.9924
a6 105.98
d6  60.61
end
end

ecce_print ecce.out

basis "ao basis" spherical print
  H library "6-31+G*"
  O library "6-31+G*"
  C library "6-31+G*"
END

dft
  mult 1
  direct
  XC b3lyp
  grid fine
  mulliken
end

driver
  default
end

task dft optimize

And run. Done!

511. When nwchem PES scans fail to constrain -- autoz failure

Another update:
My jobs have run long enough now that I can confirm that using your own z matrix still does not constrain the bond lengths i.e. the link at the end of the post is useless.

Update:
I'm not sure using a zmatrix actually solved this -- for each step in the optimization it seems that nwchem attempts to generate a new zmatrix, and probably ignoring my input (and yes, I'm using noautoz). I'll let my calcs run for a little while to see whether the constraints are honoured or not.

But I'm getting really frustrated with nwchem right now, especially since gaussian isn't having any issues running these particular jobs (there are other issues with gaussian, such as the format of the output, etc.)

Original post:
I set up PES scans in nwchem as shown in this post: http://verahill.blogspot.de/2013/08/503-relaxed-pes-scanning-in-nwchem.html

I was noticing that while almost all of my potential energy surface scans were working out just fine, some of them would...not. What would happen is that there would be no error messages, but for some reason the e.g. atom-atom distance that was defined (and defined using constant) would not remain constant during the geometry optimization in each step.

I saw this when looking trying to move an anion (9 atoms) step-wise closer to a large, negatively charged metal oxide ion (25 atoms).

I took a while to chase this down. First I though that well maybe the distances weren't really set as immutable, but were just associated with a certain force constant -- and that the anion-anion repulsion somehow overcame this. That wasn't the case.

Instead it was something that I should've paid attention to: the zmatrix generation.
If you find that for some reason the PES scan is not constrained at all, look for something along the lines of the following in your output:

NWChem Input Module
                                -------------------


                                 molecules_def2_svp
                                 ----------------

 Scaling coordinates for geometry "geometry" by  1.889725989
 (inverse scale =  0.529177249)


          ------
          auto-z
          ------
      warning. autoz generated   10 bonds for atom   24
      warning. autoz generated   10 bonds for atom   25
      warning. autoz generated   10 bonds for atom   26
      warning. autoz generated   10 bonds for atom   27
      warning. autoz generated   10 bonds for atom   28
      warning. autoz generated   10 bonds for atom   29
  autoz: The atoms group into disjoint clusters
 cluster   1:    1    2    3    4
 cluster   2:    5    6    7    8    9   10   11   12   13   14   15
                16   17   18   19   20   21   22   23   24   25   26
                27   28   29
 Connecting clusters   1   2 via atoms    3    7 r = 3.71
  autoz: regenerating connections with new bonds
      warning. autoz generated   10 bonds for atom   24
      warning. autoz generated   10 bonds for atom   25
      warning. autoz generated   10 bonds for atom   26
      warning. autoz generated   10 bonds for atom   27
      warning. autoz generated   10 bonds for atom   28
      warning. autoz generated   10 bonds for atom   29
  autoz: excessive number of variables                  2066                   81

 AUTOZ failed to generate good internal coordinates.
 Cartesian coordinates will be used in optimizations.

If that happens, cartesian coordinates will be used, and your

python
from nwgeom import *
geom = ''' 
    geometry adjust
        zcoord
            bond 1 14 %f bond constant
        end
    end 
'''

won't do anything.

The solution is to provide the coordinates as a zmatrix instead -- and that's the focus of my next post:
http://verahill.blogspot.com.au/2013/09/512-briefly-zmatrices-in-nwchem-methanol.html

Oh, and don't forget to include noautz

510. Kernel 3.11 on Debian

Here's another semi-recycled kernel building post -- the instructions are the same as for (almost) any previous version of the kernel.

There are several ways of building a kernel. The easiest (a purely subjective statement) is to use kernel-package i.e. make-kpkg. However, every now and again I see people writing that it's been deprecated.

I never spent much time with 3.10 as it caused (subjective) slow-downs on my SL410 laptop, and had issues -- at least in the early iterations -- when building the nvidia dkms module. I compiled this on a running 3.9.2-ck kernel.

So here's 3.11 -- note that this kernel too will not build the nvidia 304.88 dkms module, although it seems that 325.15 will build with a little bit of patching.

Either way, start by doing

sudo apt-get install fakeroot build-essential ncurses-dev kernel-package
mkdir ~/tmp
cd ~/tmp
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.11.tar.xz
tar xvf linux-3.11.tar.xz
cd linux-3.11/
cat /boot/config-`uname -r`>.config
make oldconfig


Compile also drivers which will not load (COMPILE_TEST) [N/y/?] (NEW) N
Kernel compression mode
> 1. Gzip (KERNEL_GZIP)
  2. Bzip2 (KERNEL_BZIP2)
  3. LZMA (KERNEL_LZMA)
  4. XZ (KERNEL_XZ)
  5. LZO (KERNEL_LZO)
  6. LZ4 (KERNEL_LZ4) (NEW)
choice[1-6?]: 1
Timer tick handling
  1. Periodic timer ticks (constant rate, no dynticks) (HZ_PERIODIC) (NEW)
> 2. Idle dynticks system (tickless idle) (NO_HZ_IDLE) (NEW)
  3. Full dynticks system (tickless) (NO_HZ_FULL) (NEW)
choice[1-3]: 2
Memory placement aware NUMA scheduler (NUMA_BALANCING) [N/y/?] (NEW) N
Simple CPU accounting cgroup subsystem (CGROUP_CPUACCT) [N/y/?] (NEW)
Group CPU scheduler (CGROUP_SCHED) [N/y/?] (NEW)
Automatic process group scheduling (SCHED_AUTOGROUP) [N/y/?] (NEW)
Choose SLAB allocator
  1. SLAB (SLAB) (NEW)
> 2. SLUB (Unqueued Allocator) (SLUB)
choice[1-2?]: 
SLUB per cpu partial cache (SLUB_CPU_PARTIAL) [Y/n/?] (NEW)
AIX basic partition table support (AIX_PARTITION) [N/y/?] (NEW)
Linux guest support (HYPERVISOR_GUEST) [N/y/?] (NEW) N
Timer frequency
  1. 100 HZ (HZ_100)
  2. 250 HZ (HZ_250) (NEW)
  3. 300 HZ (HZ_300)
> 4. 1000 HZ (HZ_1000)
choice[1-4?]: 
Enable workqueue power-efficient mode by default (WQ_POWER_EFFICIENT_DEFAULT) [N/y/?] (NEW)
Memory Hotplug (ACPI_HOTPLUG_MEMORY) [N/y/?] (NEW)
AMD frequency sensitivity feedback powersave bias (X86_AMD_FREQ_SENSITIVITY) [N/m/?] (NEW) 
Support for PCI Hotplug (HOTPLUG_PCI) [N/y/?] (NEW) 
Kernel support for scripts starting with #! (BINFMT_SCRIPT) [Y/n/m/?] (NEW)
InfiniBand media type support (TIPC_MEDIA_IB) [N/y/?] (NEW)
Network Coding (BATMAN_ADV_NC) [N/y/?] (NEW)
NETLINK: mmaped IO (NETLINK_MMAP) [N/y/?] (NEW)
NETLINK: socket monitoring interface (NETLINK_DIAG) [N/m/y/?] (NEW)
MPLS: GSO support (NET_MPLS_GSO) [N/m/y/?] (NEW)
NCI over SPI protocol support (NFC_NCI_SPI) [N/y/?] (NEW)
NFC hardware simulator driver (NFC_SIM) [N/m/?] (NEW) 
Dummy IRQ handler (DUMMY_IRQ) [N/m/y/?] (NEW) 
Generic on-chip SRAM driver (SRAM) [N/y/?] (NEW)
ME Enabled Intel Chipsets (INTEL_MEI_ME) [N/m/y/?] (NEW) 
FlashPoint support (SCSI_FLASHPOINT) [N/y/?] (NEW)
Renesas R-Car SATA support (SATA_RCAR) [N/m/?] (NEW)
Block device as cache (BCACHE) [N/m/y/?] (NEW)
Switch target support (EXPERIMENTAL) (DM_SWITCH) [N/m/?] (NEW)
Virtual netlink monitoring device (NLMON) [N/m/y/?] (NEW)
ARC devices (NET_VENDOR_ARC) [Y/n/?] (NEW)
Qualcomm Atheros AR816x/AR817x support (ALX) [N/m/y/?] (NEW)
QLOGIC QLCNIC 83XX family SR-IOV Support (QLCNIC_SRIOV) [Y/n/?] (NEW)
Renesas SuperH Ethernet support (SH_ETH) [N/m/y/?] (NEW)
SMSC LAN911x/LAN921x families embedded ethernet support (SMSC911X) [N/m/y/?] (NEW)
Realtek RTL8152 Based USB 2.0 Ethernet Adapters (USB_RTL8152) [N/m/?] (NEW) m
Atheros ath9k rate control (ATH9K_LEGACY_RATE_CONTROL) [N/y/?] (NEW) 
Atheros 802.11ac wireless cards support (ATH10K) [N/m/?] (NEW) m
Atheros ath10k PCI support (ATH10K_PCI) [N/m/?] (NEW) m
Atheros ath10k debugging (ATH10K_DEBUG) [N/y/?] (NEW)
Atheros ath10k debugfs support (ATH10K_DEBUGFS) [N/y/?] (NEW)
Atheros ath10k tracing support (ATH10K_TRACING) [N/y/?] (NEW)
rt2800usb - Include support for rt55xx devices (EXPERIMENTAL) (RT2800USB_RT55XX) [N/y/?] (NEW) y
Realtek rtlwifi family of devices (RTL_CARDS) [M/n/?] (NEW) 
Realtek RTL8188EE Wireless Network Adapter (RTL8188EE) [N/m/?] (NEW) m
CW1200 WLAN support (CW1200) [N/m/?] (NEW)
Cypress TrueTouch Gen4 Touchscreen Driver (TOUCHSCREEN_CYTTSP4_CORE) [N/m/y/?] (NEW)
IMS Passenger Control Unit driver (INPUT_IMS_PCU) [N/m/?] (NEW)
Freescale lpuart serial port support (SERIAL_FSL_LPUART) [N/m/y/?] (NEW)
Analog Devices ADT7310/ADT7320 (SENSORS_ADT7310) [N/m/y/?] (NEW) 
GMT G762 and G763 (SENSORS_G762) [N/m/?] (NEW)
National Semiconductor LM95234 (SENSORS_LM95234) [N/m/?] (NEW)
Nuvoton NCT6775F and compatibles (SENSORS_NCT6775) [N/m/y/?] (NEW)
generic cpu cooling support (CPU_THERMAL) [N/y/?] (NEW)
X86 package temperature thermal driver (X86_PKG_TEMP_THERMAL) [M/n/?] (NEW)
MEN A21 VME CPU Carrier Board Watchdog Timer (MEN_A21_WDT) [N/m/y/?] (NEW) 
ChromeOS Embedded Controller (MFD_CROS_EC) [N/m/y/?] (NEW)
Kontron module PLD device (MFD_KEMPLD) [N/m/y/?] (NEW)
Silicon Laboratories 4761/64/68 AM/FM radio. (MFD_SI476X_CORE) [N/m/?] (NEW)
System Controller Register R/W Based on Regmap (MFD_SYSCON) [N/y/?] (NEW)
TI TPS65912 Power Management chip (MFD_TPS65912) [N/y/?] (NEW)
USBTV007 video capture support (VIDEO_USBTV) [N/m/?] (NEW)
Conexant cx25821 support (VIDEO_CX25821) [N/m/?] (NEW)
Cypress firmware helper routines (CYPRESS_FIRMWARE) [N/m] (NEW)
QXL virtual GPU (DRM_QXL) [N/m/?] (NEW)
Max number of sound cards (SND_MAX_CARDS) [32] (NEW) 
Build Display HD-audio controller/codec power well support for i915 cards (SND_HDA_I915) [N/y/?] (NEW) y
M2Tech hiFace USB-SPDIF driver (SND_USB_HIFACE) [N/m/?] (NEW)
Apple infrared receiver (HID_APPLEIR) [N/m/?] (NEW)
ELO USB 4000/4500 touchscreen (HID_ELO) [N/m/?] (NEW)
Huion tablets (HID_HUION) [N/m/?] (NEW) 
Enable USB persist by default (USB_DEFAULT_PERSIST) [Y/n/?] (NEW) Y
 FUSBH200 HCD support (USB_FUSBH200_HCD) [N/m/?] (NEW)
OHCI support for PCI-bus USB controllers (USB_OHCI_HCD_PCI) [M/n/?] (NEW)
USB-Wishbone adapter interface driver (USB_SERIAL_WISHBONE) [N/m/?] (NEW) 
Infineon Modem Flashloader USB interface driver (USB_SERIAL_FLASHLOADER) [N/m/?] (NEW) 
USB Suunto ANT+ driver (USB_SERIAL_SUUNTO) [N/m/?] (NEW)
USB Physical Layer drivers (USB_PHY) [N/y/?] (NEW) 
Faraday FOTG210 USB Peripheral Controller (USB_FOTG210_UDC) [N/m/?] (NEW) 
PXA 27x (USB_PXA27X) [N/m/?] (NEW)
MARVELL PXA2128 USB 3.0 controller (USB_MV_U3D) [N/m/?] (NEW)
USB functions configurable through configfs (USB_CONFIGFS) [N/m/?] (NEW) 
LED Support for TI LP5562 LED driver chip (LEDS_LP5562) [N/m/?] (NEW)
LED Camera Flash/Torch Trigger (LEDS_TRIGGER_CAMERA) [N/m/y/?] (NEW)
QIB DCA support (INFINIBAND_QIB_DCA) [Y/n/?] (NEW) 
Mellanox Connect-IB HCA support (MLX5_INFINIBAND) [N/m/?] (NEW) 
iSCSI Extentions for RDMA (iSER) target support (INFINIBAND_ISERT) [N/m/?] (NEW)
Set system time from RTC on startup and resume (RTC_HCTOSYS) [Y/n/?] (NEW) 
Set the RTC time based on NTP synchronization (RTC_SYSTOHC) [Y/n/?] (NEW) 
 RTC used to set the system time (RTC_HCTOSYS_DEVICE) [rtc0] (NEW) 
NXP PCF2127 (RTC_DRV_PCF2127) [N/m/?] (NEW)
Synopsys DesignWare AHB DMA support (DW_DMAC_CORE) [N/m/y] (NEW)
Synopsys DesignWare AHB DMA PCI driver (DW_DMAC_PCI) [N/m/y/?] (NEW)
WIS GO7007 MPEG encoder support (VIDEO_GO7007) [N/m/?] (NEW)
DesignWare USB2 DRD Core Support (USB_DWC2) [N/m/?] (NEW)
Mediatek Bluetooth support (USB_BTMTK) [N/m/?] (NEW) 
Intel Rapid Start Technology Driver (INTEL_RST) [N/m/y/?] (NEW) 
 Intel Smart Connect disabling driver (INTEL_SMARTCONNECT) [N/m/y/?] (NEW)
 pvpanic device support (PVPANIC) [N/m/y/?] (NEW) 
Reset Controller Support (RESET_CONTROLLER) [N/y/?] (NEW) 
FMC support (FMC) [N/m/y/?] (NEW)
XFS Verbose Warnings (XFS_WARN) [N/y/?] (NEW)
Btrfs will run sanity tests upon loading (BTRFS_FS_RUN_SANITY_TESTS) [N/y/?] (NEW)
Btrfs debugging support (BTRFS_DEBUG) [N/y/?] (NEW) 
EFI Variable filesystem (EFIVAR_FS) [N/m/y/?] (NEW) 
 NFS client support for NFSv4.2 (NFS_V4_2) [N/y/?] (NEW)
Provide Security Label support for NFSv4 server (NFSD_V4_SECURITY_LABEL) [N/y/?] (NEW)
9P Security Labels (9P_FS_SECURITY) [N/y/?] (NEW) 
Wait/wound mutex debugging: Slowpath testing (DEBUG_WW_MUTEX_SLOWPATH) [N/y/?] (NEW)
torture tests for RCU (RCU_TORTURE_TEST) [N/m/y/?] (NEW)
Ring buffer startup self test (RING_BUFFER_STARTUP_TEST) [N/y/?] (NEW)
Test functions located in the string_helpers module at runtime (TEST_STRING_HELPERS) [N/m/y] (NEW)
Debug alternatives (X86_DEBUG_STATIC_CPU_HAS) [N/y/?] (NEW)
CMAC support (CRYPTO_CMAC) [M/y/?] (NEW) 
SHA256 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA256_SSSE3) [N/m/y/?] (NEW)
SHA512 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA512_SSSE3) [N/m/y/?] (NEW)
Camellia cipher algorithm (x86_64/AES-NI/AVX2) (CRYPTO_CAMELLIA_AESNI_AVX2_X86_64) [N/m/y/?] (NEW)
Serpent cipher algorithm (x86_64/AVX2) (CRYPTO_SERPENT_AVX2_X86_64) [N/m/y/?] (NEW)
LZ4 compression algorithm (CRYPTO_LZ4) [N/m/y/?] (NEW) 
LZ4HC compression algorithm (CRYPTO_LZ4HC) [N/m/y/?] (NEW)
KVM legacy PCI device assignment support (KVM_DEVICE_ASSIGNMENT) [Y/n/?] (NEW) 
VHOST_SCSI TCM fabric driver (VHOST_SCSI) [N/m/?] (NEW) 


make menuconfig

You can now enable any additional modules by navigating the menu structure. Note that most likely you don't have to enable anything in this step, but it can come in handy if there's a major transition (e.g. the way multimedia was handled changed between kernel 3.5 and 3.6) or if you want to enable a previously disabled option.

If you only want to compile modules that are currently in use (not a good idea if you want to use the same kernel on a range of computers, or have USB devices that aren't currently plugged in) you can do that by using make localmodconfig instead of make oldconfig. I wouldn't recommend it -- in most cases it won't make a faster kernel, and space and memory tends not to be much of in the way of issues these days.

Below, change -j2 to -jX, where X is the number of cores in your CPU (not cores+1 or anything funny like that. See other posts on this blog for compilation performance tests)

make-kpkg clean
time fakeroot make-kpkg -j2 --initrd kernel_image kernel_headers
sudo dpkg -i ../linux-image-3.11.0_3.11.0-10.00.Custom_amd64.deb ../linux-headers-3.11.0_3.11.0-10.00.Custom_amd64.deb

And that's it.

509. Very briefly: Send remote commands via the dropbox folder

This is probably fairly obvious to most people.

I've got a reverse ssh tunnel set up so that I can access my work computer from home. However, for the past few days I've had the connection get stuffed up on a regular basis (it doesn't get dropped, but the connection gets refused), and it frustrates me a little bit.

While a proper ssh connection is unbeatable, I would at least be able to copy files back and forth via dropbox if I only had a way of sending commands to my work computer.

And an obvious way of doing that would be to use a cronjob and a tiny bit of bash scripting. So here we go:
While we don't have to (we could just have an empty script file instead) I like the idea of testing for the presence of a specific file in the Dropbox folder, and if it exists, execute it.

Let's call the file that tests for it runremote.sh, and put it in our home folder (~/). I personally suspect that making sure that execution output and error messages get properly logged is a good thing if you're going to fly blind like this, hence the 1> and 2>

runremote.sh

if [ -e ~/Dropbox/runme.sh ]; then
    sh ~/Dropbox/runme.sh 1>> ~/Dropbox/runme.log 2>> ~/Dropbox/runme.error &
fi

Then when you want something executed, put a file called runme.sh in ~/Dropbox:

pwd
echo 'Is it working?'
cp ~/testfile.text ~/Dropbox
date

Note that any command in runme.sh is going to be run in the ~/ folder -- not in ~/Dropbox.

And set the runremote.sh file to be executed e.g. every five minutes through cron:

crontab -e

*/5 * * * * sh ~/runremote.sh

Again, you don't need to have it test for the presence of a file, but I just instinctively like the idea.

Anyway, any command you put in ~/Dropbox/runme.sh should be executed and logged within five minutes from being synced.

You CAN use sudo (echo mypassword| sudo -S ls /root )as well by providing your password in the script file, but this is obviously not terribly safe.

508. Very, very briefly: why apt-get purge exists (anecdote)

Sometimes my brain and I fall out, and we refuse to communicate. Yesterday was a day like that.

Having put my TV computer back together again (see post), I realised that I hadn't installed conky or guake on it. So I did. Having edited my default conky.conf (a bit like the one in this post) to change eth0 to eth1, I was wondering why my wireless card wasn't showing up in the ifconfig -a output. lspci showed that I had a broadcom chip, so I installed broadcom-sta-dkms. Nothing much happened, so I rebooted. And...no network card whatsoever recognised (including the ethernet one).

Face palm. I don't have a wireless pci/pci-e card on that computer. I should know -- I had just rebuilt it...

Oh well. apt-get autoremove broadcom-sta-dkms. Nothing. Reboot. Still no eth card -- also, the eth card was obviously the broadcom one (broadcom 4401). But why wasn't it working? I had uninstalled the broadcom package after all.

A bit of googling suggested loading b44 so modprobe b44 it was. Still nothing. Then a thought hit me -- ls /etc/modprobe.d/*blacklist*

And yes -- the broadcom package had installed a broadcom-wl-blacklist.conf file in /etc/modprobe.d -- and b44 was blacklisted. At this point I could either edited the file and comment out b44, delete the file entirely or purge the package and everything would be back to normal. So I did.

So remember: just because you uninstall a package (apt-get autoremove) doesn't mean that you've restore your system to the state it was before you installed the package. To at least have a chance of restoring it you need to purge your package so that settings files are removed too (apt-get purge).

507. (re)assembling a Dell C521 -- pictures

I finally got around to getting a new (well, refurbished) power supply for my old Dell C521. While the PSU worked fine, on trying to shut down the computer before it booted linux I managed to get the power button stuck. As it turns out, it's not that easy to access things on the front panel on this particular model...

So I took everything apart. EVERYTHING. Except for removing the front panel, ironically. If this post has a lesson for anyone else it's that you shouldn't be afraid to attack (gently) your desktop with a screwdriver. Computers ain't magic.

Anyway, here are some pictures from me reassembling it. For most people it will probably be more useful to look at these pictures in reverse.

Empty case. The arrow indicates where the screws for the panel go. I forget to attach them the first time I reassembled the case...

The plastic bits go to the lights on the front panel

The arrow indicates the front button thingy which I had to attack (gently) with a screw driver to get the power button to release.

The power button. Not sure why I took a picture of it, but there you go.

Front panel screws.

The (very dusty) fan swings out in the direction shown by the arrow. Because of this, you must remove the mobo before removing the fan, which you need to remove in order to remove the front panel. Not a very good design.

The mobo gets stuck in the fabric, so you might need to poke it with a narrow plastic pen or something like that in order to get it to release when removing it. Lift up this side and then slide out the mobo when dismantling.

Another odd design feature -- the heatsink must (more or less) be removed in order to reach the front panel. Also, the CPU is in dire need of a clean and some thermal paste.

The heatsink is now attached -- the plastic bit funnels the airflow from the fan at the front across the heat sink.

The PSU is now in place, but not connected

The P1 main power connector

ATX12V (labelled p2 here) in place

The HDD in place

The CD/DVD ROM in place

PCI and PCI-E in place (and nvidia graphics card and a DTV 1000S tv card)

And it's back in one piece.

506. Extracting optimized structures from a potential energy scan in nwchem

Another update:
It now dumps the energies in a file, energies.dat, as well.

Update:
some programmes, like ecce, are more picky about the xyz format than others (e.g. jmol, vmd). I've updated the code to output xyz files that ecce too can read.

Original post:
When you use scan_input() in nwchem to do a PES scan (see e.g. here: http://verahill.blogspot.com.au/2013/08/503-relaxed-pes-scanning-in-nwchem.html) you get the energies and the gradients for the optimized structures returned as the results. However, for a casual user the atomic actual coordinates is more informative.

Here's a very simple parser written in python (2.7) which extracts the optimized structures from the output file:

#!/usr/bin/python
import sys

def getrawdata(infile):
        f=open(infile,'r')
        opt=0
        geo=0
        energy=[]
        energies=[]
        struct=[]
        structure=[]
        for line in f:
                if "Total DFT" in line:
                        line=filter(None,line.rstrip('\n').split(' '))
                        energy=float(line[4])
                if 'Optimization converged' in line:
                        opt=1
                if opt==1 and 'Geometry' in line:
                        geo=1
                if      'Atomic Mass' in line and (opt==1 and geo==1):
                        opt=0
                        geo=0
                        struct+=[structure]
                        energies+=[energy]
                        structure=[]
                if opt==1 and geo==1:
                        structure+=[line.rstrip()]
        return struct,energies

def genxyzstring(coords,element):
        x_str='%10.5f'% coords[0]
        y_str='%10.5f'% coords[1]
        z_str='%10.5f'% coords[2]
 
        xyz_string=element+(3-len(element))*' '+10*' '+\
        (8-len(x_str))*' '+x_str+10*' '+(8-len(y_str))*' '+y_str+10*' '+(8-len(z_str))*' '+z_str+'\n'
 
        return xyz_string

def getstructures(rawdata):
        
        n=0
        for structure in rawdata:
                
                n=n+1
                num="%03d" % (n,)
                g=open('structure_'+num+'.xyz','w')
                itson=False
                cartesian=[]
                        
                for item in structure:
                        
                        if itson and not(item==""):
                                coords=filter(None,item.split(' '))
                                coordinates=[float(coords[3]),float(coords[4]),float(coords[5])]
                                element=coords[1]
                                cartesian+=[genxyzstring(coordinates,element)]
                                #cartesian+=[coords[1]+'\t'+coords[3]+'\t'+coords[4]+'\t'+coords[5]+'\n']
                
                        if "---" in item:
                                itson=True
                        if item=="" and itson==True:
                                itson=False
                                if not(len(cartesian)==0):
                                        g.write(str(len(cartesian))+'\n')
                                        g.write('Structure '+str(n)+'\n')
                                        for line in cartesian:
                                                g.write(line)
                                        g.close()
                                cartesian=[]
        return 0
        
if __name__ == "__main__":
        infile=sys.argv[1]
        rawdata,energies=getrawdata(infile)
        structures=getstructures(rawdata)

        g=open('energies.dat','w')
        for n in range(0,len(energies)):
                g.write(str(n)+'\t'+str(energies[n])+'\n')
        g.close()

Presuming that you've saved it as pes_parse.py you can then generate a series of xyz files with the structures, catenate them into a trajectory file, and open it in e.g. jmol. I'm using the output from example 1 in http://verahill.blogspot.com.au/2013/08/503-relaxed-pes-scanning-in-nwchem.html as the example:

chmod +x pes_parse.py
./pes_parse.py nwch.nwout
ls

nwch.nwout    structure_001.xyz  structure_003.xyz  structure_005.xyz  structure_007.xyz  structure_009.xyz  structure_011.xyz  structure_013.xyz  structure_015.xyz  structure_017.xyz  structure_019.xyz
pes_parse.py  structure_002.xyz  structure_004.xyz  structure_006.xyz  structure_008.xyz  structure_010.xyz  structure_012.xyz  structure_014.xyz  structure_016.xyz  structure_018.xyz

cat structure_*.xyz >> trajectory.xyz 
jmol trajectory.xyz

You can go through the structures by clicking on the arrows indicated by the white arrow:

Finally, using VMD it's easy to make videos -- note that they for some reason look awful here (seems like a lot of frames are removed, in particular from the beginning of the run):

And here's the SN2 reaction from post 503: