Saturday, August 01, 2020

SOLR: Non English (Latin) Characters in Field Name

The SOLR documentation mentions following requirement for defining name of a field.


name
The name of the field. Field names should consist of alphanumeric or underscore characters only and not start with a digit. . . .
While working on a dictionary website, the JSON documents that I created had field names in Hindi. After indexing the data I was surprised to see that field names in the data were converted to multiple underscore letters e.g. field name शब्द was converted to ____. According to SOLR documentation शब्द should have been allowed as field name.

Looks like SOLR developers have assumed that only 26 letters in Latin script are alphabets. Mentioning this assumption explicitly in documentation would have been helpful.

After a closer scrutiny of solrconfig.xml file, I found following configuration, which converts anything that is not Latin alphanumeric in field name to underscore while indexing the data.

<updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating">
   <str name="pattern">[^\w-\.]</str>
   <str name="replacement">_</str>
</updateProcessor>

Changing the pattern regex for FieldNameMutatingUpdateProcessorFactory to something like below will allow SOLR to accept non Latin alphabets in field name.

   <str name="pattern">[\s]</str>

[\s] is a too generic pattern to use in real life scenario. This pattern should be further restricted to a limited set of characters that one intend to use in field name.

Monday, May 25, 2020

How to customize Moodle Mobile App?

Chinese Virus (COVID-19) has created unimaginable chaos that no one could have thought about. As a side effect most of us have forced spare time to share with family at home. I utilized a part of this time to explore something new and that is Moodle.

Moodle is a widely popular Learning Management System and it has a Mobile app to deliver the content to complement the web deliver channel. Installation of Moodle and setting up few courses was a fairly simple activity. Sample courses were rendering well on standard Moodle App for Android.

Then I decided customize the mobile app to give it a unique branding. I was expecting it to be a simple task but it came out to be a tough activity. My lack of previous experience in mobile app development was a big contributor to that. Moodle for Mobile forum was not of much help. After putting lots of effort I was able to get it done. I have documented the steps required to build and customize the Moodle App.

Environment Setup

Moodle has a good document to setup the local development environment. Though it is bit outdated at places but following that I was able to setup the development environment on my Mac.

Now clone the source code from https://github.com/moodlehq/moodleapp

Configuration

Next step is to change the configuration to give it a custom branding. Here is the list of files and the relevant configuration properties that needs to be changed.

  • Change following properties in config.xml.

    • Widget id

    • Widget version

    • App name

    • Description

    • Author

    • AppendUserAgent

    • URL_SCHEME (change it only if plan to block official Moodle App from accessing your website)

      • Update server side property “tool_mobile | forcedurlscheme” at the above-mentioned URL.

  • Make following changes in src/config.json

    • app_id

    • appname

    • desktopname

    • versioncode

    • versionname

    • Empty demo_sites element

    • customurlscheme (related to URL_SCHEME)

    • siteurl

    • sitename

    • privacypolicy

  • Update custom url scheme in package.json (related to URL_SCHEME)

Logo & Splash Screen

Having own logo and splash screen is what people want in custom moodle app. Below is the list of images for logo and splash screen that you have to put in source code.

  • resources/icon.png

  • resources/splash.png

  • resources/icon/icon-background.png

  • resources/icon/icon-foreground.png


Following images should be removed, so that above-mentioned images are used during build process.


  • resources/splash.png.md5

  • resources/android/icon.png

  • resources/android/icon.png.md5

  • resources/android/icon-foreground.svg

  • resources/ios/icon.png

  • resources/ios/icon.png.md5


Now run ionic cordova resources command to generate icons and splash images for various sized screens. Successful execution of this command will print logs as shown below.


IMUL-ML0245:moodleapp vinodsingh$ ionic cordova resources

> cordova-res

[cordova-res] Generated 24 resources for android

[cordova-res] Generated 50 resources for ios

[cordova-res] Wrote to config.xml


  ──────────────────────────────────────


     Update available: 5.2.3 → 5.4.16

       Run npm i -g ionic to update


  ──────────────────────────────────────

Note:- In my case it did not created 4 android icons whose name were ending in *-smallicon.png. I removed those images and deleted corresponding entries from config.xml.


Very informative documentation about creating icon and splash images is available here -

https://ionicframework.com/docs/cli/commands/cordova-resources

https://ionicframework.com/blog/automating-icons-and-splash-screens/

Firebase Project

For every Android app a Firebase project is madatory. Go to Firebase Console and create a new project and add the app to that. Download the google-services.json file and place it at the root folder.

Build - Debug

Now all required changes are done and we are ready to build a debug version of the app. Run ionic cordova build android to build the app with debug messages. It may fail with the following error.


Using cordova-fetch for cordova-android@8.0.0

Adding android project...

Creating Cordova project for the Android platform:

Path: platforms/android

Package: bharat.sarathi.mobile

Name: _____.____

Activity: MainActivity

Android target: android-28

Subproject Path: CordovaLib

Subproject Path: app

Android project created with cordova-android@8.0.0

Unable to graft xml at selector "/manifest/uses-feature[@android:name='android.hardware.location.gps']" from "/Users/vinodsingh/code/GitHub/moodleapp/platforms/android/app/src/main/AndroidManifest.xml" during config install

[ERROR] An error occurred while running subprocess cordova.

        

        cordova platform add android --save exited with exit code 1.

        

        Re-running this command with the --verbose flag may provide more information.


Running same command with --verbose provides precise error message as shown below.


[12:31:35] Local gulp not found in ~/code/GitHub/moodleapp

[12:31:35] Try running: npm install gulp


After installing local gulp reran the ionic cordova build android --verbose command and it succeeded. The APK file will be placed here - platforms/android/app/build/outputs/apk/debug/app-debug.apk. Debug app can be tested on an android emulator.


Running the build command again may fail with following error -


Users/vinodsingh/code/GitHub/moodleapp/platforms/android/app/src/main/AndroidManifest.xml:33:5-66 Error:

Element uses-feature#android.hardware.location.gps at AndroidManifest.xml:33:5-66 duplicated with element declared at AndroidManifest.xml:32:5-91

/Users/vinodsingh/code/GitHub/moodleapp/platforms/android/app/src/main/AndroidManifest.xml Error:

Validation failed, exiting


To fix this error remove the below line from AndroidManifest.xml file and run the build command again.


    <uses-feature android:name="android.hardware.location.gps" android:required="false" />

Intent Filters

Testing revealed an issue with the app. After authentication in browser control does not go back to the app. User is stuck in the browser itself. This issue makes app unusable. After lots of struggle, I figured out that the following intent-filters are missing from main activity in AndroidManifest.xml.


    <intent-filter>

        <action android:name="android.intent.action.VIEW" />

        <category android:name="android.intent.category.DEFAULT" />

        <category android:name="android.intent.category.BROWSABLE" />

        <data android:scheme="URL_SCHEME" />

    </intent-filter>

    <intent-filter>

        <action android:name="android.intent.action.VIEW" />

        <category android:name="android.intent.category.DEFAULT" />

        <category android:name="android.intent.category.BROWSABLE" />

        <data android:host=" " android:pathPrefix="/" android:scheme=" " />

    </intent-filter>


Ideally they should have been there due to the inclusion of customurl-plugin. Adding these intent-filters manually fixed the issue.

Build - Release

After successful testing the app on emulator, I am ready for production build and test on a real device. Reading ionic publishing document is recommended before proceeding with next steps.

Create key [one time]

We have to create a key for signing the app. The Java keytool command as shown below can generate the required key.

keytool -genkey -v -keystore my-release-key.keystore -alias alias-name -keyalg RSA -keysize 2048 -validity 10000

Build

Moodle suggests few edits to the source code to make it production ready. Suggested code changes are documented here. After making code changes run the build command to create APK without debug messages.


ionic cordova build android --prod --release


It will create a apk file at location - platforms/android/app/build/outputs/apk/release/app-release-unsigned.apk location.

Sign the APK

Now this release apk file needs to be signed with key that we generated a while back.


jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore my-release-key.keystore app-release-unsigned.apk alias-name


Next is to zip align the APK using zipalign tool available in Android SDK.


~/Library/Android/sdk/build-tools/29.0.3/zipalign -v 4 app-release-unsigned.apk my-production-ready.apk


Now APK is similar to what one will download from the Play Store. It is ready for testing on an emulator or real Android device.

Wednesday, September 07, 2011

Why NoSQL is in fashion?

Couple of years back NoSQL was relatively unheard term. It has become a new buzzword in town, catching people's imagination. Almost all big daddies of Internet (Google, Facebook, Yahoo, Twitter, Amazon, Linked-in etc.) are using NoSQL at massive scale and many of them have their own NoSQL products. Every month there is a launch of new NoSQL product catering to some specific use cases / requirements. So what are the reasons behind NoSQL's rise to popularity?

Background

RDBMS and SQL has ruled the world for last 3 decades. They were (are) the defacto standard for storing business data. In fact there was no alternative to RDBMS and SQL combination. When RDBMS were developed the disk space was scarce, hence a lot of thought has gone around saving precious disk space. Development in hardware technologies have made available literally unlimited storage space at lower prices. Evolution of Internet in last decade has paved way for altogether different kind of applications and use cases, which were never thought earlier. Web2 applications (say social networking) have entirely different requirements than typical enterprise applications. Other than that Internet has changed the way application developers think about the data and its usage & storage.

Requirement

Nowadays enormous amount of data is being produced everyday and the rate at which data is produced is also increasing. Data size is so huge that organizations spread it across data center as it does not fit in a single rack or data center. Though data availability is also an equally important factor in using multiple data centers for storage. Applications have to available 24x7 no matter what. Data has to survive the data center failure as well, forget about failure of a disk. Instead of archiving data, Facebook type of applications keep it online forever.

Social networking sites are coming up with new features quite frequently and sometimes rolling back them. At times their new features were never thought about earlier. They don't want to be confined by the schema constraints of RDBMS which is quite difficult (though not impossible) to change. Support for evolving schema has become a necessity and NoSQL solutions are a right fit there instead of RDBMS.

Data size, flexible schema and availability are the 3 key factors where traditional RDBMS do not perform well after a certain limit.

Thought Process

Web2 applications has changed how people think about application architecture. It is an altogether different thought process then what it was few years back.

While designing an enterprise application we used to think about "how to store the data" but today the primary thought is "how we will use the data". This is a paradigm shift. Data storage is designed based on how that is going to be used, leading to optimized performance for a given requirement at times at the cost of flexibility.

"Referencing" entities was a wonderful feature in RDBMS but for large scale web 2 application that does not matter at all. For them instead of referencing "embedding" is the killer feature. Storing related data (say personal detail and address) together makes sense as they don't have any meaning independently. This is similar to denormalization in RDBMS.

For many features of web 2 applications ACID transactions are also not a primary requirement. Say if a tweet does not gets published many users won't mind to type that again and hit the enter button as long as that does not happen frequently. If avoiding ACID transactions increases the throughput many fold for not so critical features, what else you want :-). It is similar to old MySQL though with much larger data-set than what MySQL can support.

Conclusion

There are several use cases where NoSQL perform much better and it makes perfect sense to use NoSQL instead of trying to scale the RDBMS by using expensive methods. RDBMS’s are really good at doing what they do, which is storing flat, relational, tabular data in a consistent manner and getting wonderful reports out of that data. They still remain the best solution for storing relational data.While NoSQL databases are good at performance, availability, schema-less persistent etc., which has become a basic requirement for today's applications. Support for these features is the main reason for meteoric rise to fame of the NoSQL.

Thursday, April 21, 2011

Blessed with a baby girl

It has been an year since I blogged last time. last one year was very hectic for me both at workplace and home. There was tons of workload at office at times eating into weekends. We were expecting a baby, so I have to take additional responsibilities at home as well. In December we were blessed with a gorgeous baby girl. That was the most fulfilling moment of our life. It has become an everyday celebration for our son. For quite long time he was dreaming of having a brother or sister.



Next month I will be joining a new workplace, which is closer to my home. That will spare some precious time to spare with family.

Hopefully I will be back to blogging soon.

Monday, April 12, 2010

JAX-WS Web service with Maven

One of my earlier post Building JAX-WS Webservice was using Ant as build tool for developing web services. With each passing day Maven is gaining more popularity and became preferred build tool for many developers. I used to get emails from several readers about working JAX-WS examples with Maven as build tool. In this post I will try to demonstrate to publish and consume a JAX-WS web service using Maven.

The source code used in this post is available here, which will be referred at several places in this post.

The jaxws-maven-plugin has two goals wsgen and wsimport.

  • wsgen - This reads a service endpoint implementation class and generates all of the portable artifacts for a JAX-WS web service. With newer versions (tested with 2.2) of JAX-WS, execution of this task is not required for publishing webservice
  • wsimport - This tool reads WSDL and generates client side artifacts. We will be using this goal for generating client and consuming a web service.

Publish Web service

For publishing a web service we need to write an interface and its implementation then annotate them with relevant annotations. The JAX-WS libraries should be added as dependency in pom.xml. A sample (partial) build script is shown below-

<dependencies>
    <dependency>
        <groupId>com.sun.xml.ws</groupId>
        <artifactId>jaxws-rt</artifactId>
        <scope>compile</scope>
        <version>2.2</version>
    </dependency>
</dependencies>
. . .
<build>
    <plugins>
        <plugin>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>maven-jetty-plugin</artifactId>
            <configuration>
                <scanIntervalSeconds>10</scanIntervalSeconds>
                <connectors>
                    <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                        <port>8080</port>
                        <maxIdleTime>60000</maxIdleTime>
                    </connector>
                </connectors>
            </configuration>
        </plugin>
    </plugins>
</build>

We can see in the above Maven script that the wsgen is not required. The JAX-WS implementation will take care of that at runtime. To make testing easier I have used Jetty plugin, which will publish the web service for further consumption by the client in next step. To publish the web service invoke following command in 'webservice' directory of the source code of this post-
mvn -e clean compile jetty:run

Create Client

For creating web service client the wsimport goal of jaxws-maven-plugin will be used. The wsimport goal creates portable client artifacts by parsing a WSDL. In this example the WSDL URL is the one which was published by starting the web service at previous step. The pom.xml of client side application will look like below-

<dependencies>
    <dependency>
        <groupId>com.sun.xml.ws</groupId>
        <artifactId>jaxws-rt</artifactId>
        <scope>compile</scope>
        <version>2.2</version>
    </dependency>
</dependencies>
. . .
<build>
    <plugins>
        <!-- Generate client using WSDL -->
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>jaxws-maven-plugin</artifactId>
            <configuration>
                <packageName>wsclient</packageName>
                <wsdlUrls>
                    <wsdlUrl>http://localhost:8080/webservice/myService?wsdl</wsdlUrl>
                </wsdlUrls>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>1.1</version>
            <executions>
                <execution>
                    <goals>
                        <goal>java</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <mainClass>client.WsClient</mainClass>
            </configuration>
        </plugin>
    </plubins>
</build>

For the sake of simplicity I have used exec-maven-plugin here, which will run the test class to invoke the web service from build script itself. To generate client and run the test class, trigger the build from 'webservice-client' directorty using following command-
mvn -e clean compile exec:java


The examples shown above were tested with following configuration-
  • Java 1.6.18
  • JAX-WS 2.
  • Maven 2.2.1
  • Windows XP

Note:- If JAX-WS 2.2 is used with older versions of Java 6 (e.g. Java 1.6.03) then you may encounter errors like shown below-
SEVERE: WSSERVLET11: failed to parse runtime descriptor: java.lang.LinkageError: JAXB 2.1 API is being loaded from the bootstrap classloader, but this RI (from jar:file:/C:/Documents%20and%20Settings/Vinod%20Singh/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2/jaxb-impl-2.2.jar!/com/sun/xml/bind/v2/model/impl/ModelBuilder.class) needs 2.2 API. Use the endorsed directory mechanism to place jaxb-api.jar in the bootstrap classloader. (See http://java.sun.com/j2se/1.6.0/docs/guide/standards/)
java.lang.LinkageError: JAXB 2.1 API is being loaded from the bootstrap classloader, but this RI (from jar:file:/C:/Documents%20and%20Settings/Vinod%20Singh/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2/jaxb-impl-2.2.jar!/com/sun/xml/bind/v2/model/impl/ModelBuilder.class) needs 2.2 API. Use the endorsed directory mechanism to place jaxb-api.jar in the bootstrap classloader. (See http://java.sun.com/j2se/1.6.0/docs/guide/standards/) at com.sun.xml.bind.v2.model.impl.ModelBuilder.(ModelBuilder.java:173)

The reason of these errors is that the earlier versions of Java 6 are using lower versions of JAX-WS while we are trying to use higher version of JAX-WS. To overcome these errors either we need to downgrade the JAX-WS version in our runtime environment or use endorsed directory mechanism.

Hope this will be useful to some of the readers.

Thursday, February 25, 2010

m2eclipse plugin makes M2_REPO variable non modifiable in Eclipse

Yesterday for first time I installed m2eclipse plugin and all of a sudden projects failed to compile. I was using Maven for quite long time hence adding all library dependencies for a project using a variable named M2_REPO, which was pointing towards a location where Maven downloaded jars were stored. After installation of m2eclipse plugin Eclipse was not able to resolve the absolute path of the jars in classpath.



The value of the M2_REPO variable was changed itself to some path other than what I configured. When I tried to fix that, to my surprise I can't edit it any longer. It was marked with this text "M2_REPO (non modifiable)". After poking here and there for a while I come to know that m2eclipse picks the value of M2_REPO variable from $MAVEN_HOME/conf/settings.xml (MAVEN_HOME is the currently selected Maven installation) file using <localRepository> element-

<localRepository>PATH_TO_REPOSITORY</localRepository>
If nothing is specifed in settings.xml then ${user.home}/.m2 directory becomes the value of M2_REPO variable.

Wednesday, February 17, 2010

Remove old images of Linux (Ubuntu) kernel

Development of Linux is quite fast paced and new versions of kernel are released pretty frequently. I have a dual boot desktop with Ubuntu and Windows XP. With each update grub boot menu keeps on growing and I have to make Windows XP default OS as desktop is used by my 8 year son as well. Each version of kernel takes up > 100MB space, though that is not a big worry with huge storage available at much cheaper price.

The unused (older) versions of kernel can be removed safely. Usually I prefer to keep the last version along with "last - 1", to be on safer side so that I have something to fallback upon if something goes wrong with the latest kernel. To remove a version of kernel go to Synaptic Package Manager and search for linux-image and select the ones you want to remove.



You'll get an option to remove completely (Mark for Complete Removal), that will get rid of everything including configuration files used by that package. Before removing kernel make sure to not remove the current version as that will make the system unusable. Use uname -a command to know the currently used version of kernel. Output of this command looks like below on my machine-

vinod@vinod-desktop:~$ uname -a
vinod@vinod-desktop:~$ Linux vinod-desktop 2.6.31-19-generic #56-Ubuntu SMP Thu Jan 28 02:39:34 UTC 2010 x86_64 GNU/Linux

You can also find and remove linux-headers using this simple method.

Monday, January 11, 2010

Share a datasource among several applications in Tomcat using Atomikos

In a previous post I discussed about how to use Atomikos JTA implementation in Tomcat. We had a requirement of sharing a globally defined datasource among several applications deployed in Tomcat container. The datasource lookup from the one application (whichever does the lookup first) succeeds but it always fails from the second application with an error message like-

Another resource already exists with name DATA_SOURCE_NAME - pick a different name

Why? We were using the sample bean factory available at Atomikos' site, which tries to recreate the datasource on each lookup. Within an application one can cache the datasource after initial JNDI lookup call. But what about doing a lookup of same resource from multiple applications?

After poking around for a while in Atomikos source code I found that after creation, datasource instance is cached by the Atomikos. With a minor modification in bean factory as shown below, the cached instance can be retrieved instead of recreating it.
// see if this DataSource is already initialized then return cached object
// if available, else Atomikos will throw an exception
try {
    bean = IntraVmObjectRegistry.getResource(((AbstractDataSourceBean) bean).getUniqueResourceName());
    if (log.isInfoEnabled())
        log.info("Returning cached value of AbstractDataSourceBean (Atomikos): " + bean);

    return bean;
} catch (NameNotFoundException nfe) {
    // OK, it is not available go ahead and create one
}
With this change a globally defined datasource can be used by multiple applications inside a Tomcat container to use Atomikos JTA implementation.

Wednesday, December 23, 2009

Start Apache as service in Linux (Fedora)

The Apache HTTP server is installed by default in all Linux distributions and it is configured to run as service. I usually prefer to install Apache from source instead of using the default one. After installing Apache I want to run that instance as service so that it can be restarted automatically whenever machine is rebooted. This can be achieved easily by modifying /etc/rc.d/init.d/httpd file, which is configured to run the Apache installed with OS.

Comment the following section, though it is not required but I prefer to keep all configuration at one place


if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi
Modify the following lines-
apachectl=/usr/sbin/apachectl
httpd=${HTTPD-/usr/sbin/httpd}
prog=httpd
pidfile=${PIDFILE-/var/run/httpd/httpd.pid}
to point towards your Apache installation
apachectl=/opt/apps/httpd-2.2.14/bin/apachectl
httpd=${HTTPD-/opt/apps/httpd-2.2.14/bin/httpd}
prog=httpd
pidfile=${PIDFILE-/opt/apps/httpd-2.2.14/logs/httpd.pid}
Now your custom Apache instance will be started by OS instead of the default one.

Thursday, December 10, 2009

Send XML data over web service

Occasionally I am being asked a common question, "How to send XML data over web service?". In Java (in fact in any programming language) XML is just a string (sequence of characters). So sending XML as request parameter or receiving it in response is just like handling any other string data.

For the following XML data from web service client and server-

XML from client
<name>vinod</name>

XML from server
<greeting>Hello! vinod</greeting>

This is the web service (JAX-WS) request and response-
REQUEST
<?xml version="1.0" ?>
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
    <S:Body>
        <ns2:xmlData xmlns:ns2="http://vinodsingh.com">
            <data>&lt;name&gt;vinod&lt;/name&gt;</data>
        </ns2:xmlData>
    </S:Body>
</S:Envelope>

RESPONSE
<?xml version='1.0' encoding='UTF-8'?>
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
    <S:Body>
        <ns2:xmlDataResponse xmlns:ns2="http://vinodsingh.com">
            <return>&lt;greeting>Hello! vinod&lt;/greeting></return>
        </ns2:xmlDataResponse>
    </S:Body>
</S:Envelope>

The above example clearly shows that JAX-WS runtime handles XML strings pretty well. The source code available here includes an example of sending bigger XML data.

Tuesday, December 01, 2009

JTA Transactions with Atomkios in Tomcat

Tomcat is the leading Java Servlet container. Many of our applications do not need full blown Java EE Application Server and a Servlet container like Tomcat is more than enough to run them even in production. Some of our applications need JTA transactions capabilities, which are not provided by Tomcat out of the box. There are very few open source JTA implementations. JOTM used to be most popular among them, unfortunately its development is dead for last several years though it seems to getting revived lately at its new home. Atomikos is another JTA implementation, which was open sourced (Transactions Essentials) when JOTM went into hibernation.

Integrating Atomikos with Tomcat is fairly easy, it can be done by following simple steps given below-

Create a configuration file for Atomikos at $TOMCAT_HOME/lib/transactions.properties location with contents like below-

com.atomikos.icatch.service=com.atomikos.icatch.standalone.UserTransactionServiceFactory
com.atomikos.icatch.automatic_resource_registration=true
com.atomikos.icatch.output_dir=../work
com.atomikos.icatch.log_base_dir=../work
com.atomikos.icatch.enable_logging=true
com.atomikos.icatch.console_log_level=INFO

Define a DataSource in $TOMCAT_HOME/conf/context.xml as shown below-

<Transaction factory="com.atomikos.icatch.jta.UserTransactionFactory" />

<Resource name="jdbc/pgDS"
                auth="Container"
                type="com.atomikos.jdbc.AtomikosDataSourceBean"
                factory="com.vinodsingh.atomikos.tomcat.DataSourceBeanFactory"
                uniqueResourceName="jdbc/pgDS"
                xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
                xaProperties.serverName="localhost"
                xaProperties.portNumber="5432"
                xaProperties.databaseName="test"
                xaProperties.user="test"
                xaProperties.password="test"
                maxPoolSize="3"
                minPoolSize="1" />

Copy following files to $TOMCAT_HOME/lib directory-
  • JDBC driver of the Database being used.
  • JTA API (jta-1.1.jar).
  • Atomikos Essentials (transactions-essentials-all-3.5.9.jar), name may change according to the version being used.
  • Tomcat integration jars available at Atomikos' site or atomikos-tomcat.jar built from the source code attached with this post.
All these changes will make UserTransaction object available in Tomcat's JNDI tree and it can be used in same way as we do in any regular Java EE application server. An example of using UserTransaction is shown in following code snippet-
public String insertRecord(String value) {
    UserTransaction utx = null;
    try {
        utx = ServiceLocator.getTrasactionManager();
        utx.begin();
        // Do transactional work
        pgDAO.insertRecord(value);

        // commit the transaction
        utx.commit();
    } catch (Exception e) {
        try {
            if (utx != null)
                utx.rollback();

            String msg = "Oops! Transaction is rolled back due to an exception";
            log.error(msg, e);
            return msg;
        } catch (Exception e1) {
            log.error("Failed to rollback the transaction", e1);
            return "Failed to rollback the transaction" + e1.getMessage();
        }
    }
    return "Transaction succeded";
}
To see transactions in action compile the attached source code, deploy the generated jta.war on Tomcat and open http://localhost:8080/jta/ URL in your browser. Thats all it takes to integrate Atomikos JTA transactions in Tomcat.

Note:-
  1. Though Atomikos' documentation about Tomcat 6 integration talks about configuring AtomikosLifecycleListener in $TOMCAT_HOME/conf/server.xml as well. But Atomikos seems to work well inside Tomcat without that configuration.
  2. Here I have slightly modified the code for Tomcat integration from what is available at Atomikos website, more details about the same will follow in upcoming posts.

Source code for this post is available here.

Sunday, October 04, 2009

Migrating from Bugzilla to Redmine

It has been a while since I started evaluating Redmine and later started using as well. Earlier we were using Bugzilla for issue tracking, which contains good amount of data (current and historical as well). This data can't be left behind just for the sake of using some new tool. I was looking at ways to migrate this data. Robert Heath's PHP script for migration was the best solution I found.

His scripts deletes all Redmine data and migrates everything from Bugzilla to Redmine, which I did not want and also we have a highly customized instance of Bugzilla, so using that script out of box was not possible. I wanted to pick and choose the data for migration instead of dumping everything in Redmine. The Bugzilla was having internal authentication for users while we wanted to have a mix of LDAP and internal authentication in Redmine.

To meet all these (and many more) requirements, big changes were required to Robert's script. Being a long time Java programmer, I hardly have any knowledge of PHP. Making changes to Robert's PHP script was difficult to me, so I rewrote the script again in Java. I used Robert's script as a reference to map Redmine and Bugzilla types and for SQL scripts. Without that it would have been quite difficult to write the script.

Though it took some effort to write the migration script but the effort spent was worth it. Now with Redmine we have a consolidated view of issues and source code. Migration has been seamless and users are much happier with Redmine. Now I am looking at possibilities about writing few plugins for Redmine to customize it for some innovative usages.

Update Feb 13, 2010: As I was being asked for the script frequently, here is the code I used for migration. This may not work against each and every Bugzilla installation, modify it according to your needs.

/**
 *
 * 1. All internal IDs are associated as Developer (4) to projects, while external as
 * Client (6).
 *
 */
public class Migrator {

    private Log log = LogFactory.getLog(this.getClass());
    String filePath = "D:\\bugzillaAttachments"; // where to store bugzilla attachments
    SimpleDateFormat sdf = new SimpleDateFormat("yyMMddhhmmss");
    private String projectList = "1, 2, 3, 4, 5";
    private static Map priority = new HashMap();
    private static Map status = new HashMap();
    private static Map tracker = new HashMap();
    // old values are key and new ones values
    private Map userMap = new HashMap();
    private Map existingUserMap = new HashMap();
    private Map projectMap = new HashMap();
    private Map categoryMap = new HashMap();
    private Map issueMap = new HashMap();
    // key > product_id + version name
    private Map versionMap = new HashMap();

    static {
        priority.put("P1", 5);
        priority.put("P2", 4);
        priority.put("P3", 3);
        priority.put("P4", 2);
        priority.put("P5", 1);

        status.put("UNCONFIRMED", 1);
        status.put("NEW", 1);
        status.put("ASSIGNED", 2);
        status.put("REOPENED", 7);
        status.put("RESOLVED", 3);
        status.put("VERIFIED", 5);
        status.put("CLOSED", 5);

        tracker.put("trivial", 1);
        tracker.put("minor", 1);
        tracker.put("normal", 1);
        tracker.put("major", 1);
        tracker.put("critical", 1);
        tracker.put("blocker", 1);
    }

    public Migrator() {
        Connection destCon = null;
        Connection srcCon = null;
        try {
            Class.forName("org.postgresql.Driver");
            destCon = DriverManager.getConnection("jdbc:postgresql://redmine_server:5432/redmine", "user",
                            "pwd");
            destCon.setAutoCommit(false);
            log.info("PostgreSQL/Redmine connection opened");

            Class.forName("com.mysql.jdbc.Driver");
            srcCon = DriverManager.getConnection("jdbc:mysql://bugzilla_server:3306/test", "user", "pwd");
            log.info("MySql/Bugzilla connection opened");

            loadExistingUsers(destCon);
            createUsers(destCon, srcCon); // create users
            createProjects(destCon, srcCon); // create projects
            destCon.commit();
        } catch (Exception e) {
            try {
                destCon.rollback();
                log.info("Rolled back changes");
            } catch (Exception e3) {
                log.error("Unable to rollback", e3);
            }
            log.error("", e);
        } finally {
            try {
                destCon.close();
                log.info("PostgreSQL/Redmine connection closed");
                srcCon.close();
                log.info("MySql/Bugzilla connection closed");
            } catch (Exception e2) {
                log.error("Error while closing connections", e2);
            }
        }
    }

    public static void main(String[] args) {
        new Migrator();
    }

    // load existing users and don't recreate them
    private void loadExistingUsers(Connection destCon) throws Exception {
        String SELECT = "SELECT mail, id FROM users";
        Statement stmt = null;
        ResultSet rs = null;
        try {
            stmt = destCon.createStatement();
            rs = stmt.executeQuery(SELECT);
            while (rs.next()) {
                existingUserMap.put(rs.getString(1), rs.getInt(2));
            }
        } finally {
            close(rs);
            close(stmt);
        }
    }

    private void createUsers(Connection destCon, Connection srcCon) throws Exception {
        String SELECT = "SELECT userid, login_name, realname, disabledtext FROM profiles "
                        + "WHERE userid IN (SELECT ugm.user_id FROM user_group_map ugm, group_control_map gcm "
                        + " WHERE ugm.group_id = gcm.group_id AND gcm.product_id IN (" + projectList + "))";
        String INSERT = "INSERT INTO users (login, mail, firstname, lastname, language, status, auth_source_id, created_on, updated_on) "
                        + "VALUES (?, ?, ?, ?, 'en', ?, ?, now(), now())";
        log.info("Creating uses . . .");
        Statement selectStmt = null;
        PreparedStatement insertStmt = null;
        ResultSet selectRs = null;
        ResultSet keysRs = null;
        try {
            insertStmt = destCon.prepareStatement(INSERT, new String[] { "id" });
            selectStmt = srcCon.createStatement();
            selectRs = selectStmt.executeQuery(SELECT);
            while (selectRs.next()) {
                int userId = selectRs.getInt(1);
                String loginName = selectRs.getString(2).trim();
                // don't recreate an existing user
                if (existingUserMap.containsKey(loginName)) {
                    userMap.put(userId, existingUserMap.get(loginName));
                    log.info("Bugzilla user [" + userId + ", " + loginName
                                    + "] > already exists as Redmine user [" + existingUserMap.get(loginName)
                                    + "]");
                    continue;
                }
                String realName = selectRs.getString(3).trim();
                String disabledText = selectRs.getString(4);

                // Keep non internal user ids with full email address, this will enable
                // future employees with same id
                String newLoginName;
                if (loginName.endsWith("@domain.com")) {
                    newLoginName = loginName.substring(0, loginName.indexOf("@"));
                    if (newLoginName.length() > 20) {
                        // Our LDAP (AD) has a limit of 20 chars
                        newLoginName = loginName.substring(0, 20);
                    }
                } else {
                    newLoginName = loginName;
                }

                String firstName;
                String lastName;
                if (realName.contains(" ")) {
                    firstName = realName.substring(0, realName.indexOf(' '));
                    lastName = realName.substring(realName.lastIndexOf(' ') + 1);
                } else {
                    // in few cases 'realName' is email address
                    if (realName.contains("@"))
                        realName = realName.substring(0, loginName.indexOf("@"));

                    firstName = realName.length() <= 30 ? realName : realName.substring(0, 30);
                    lastName = "";
                }
                // insert the user in Redmine
                insertStmt.setString(1, newLoginName);
                insertStmt.setString(2, loginName);
                insertStmt.setString(3, firstName);
                insertStmt.setString(4, lastName);
                // Wherever 'disabledtext' contains something mark his status as 3 (lock)
                if (disabledText != null && disabledText.trim().length() > 0)
                    insertStmt.setInt(5, 3);
                else
                    insertStmt.setInt(5, 1);
                // If user id is not @domain.com, mark their auth source as internal
                if (loginName.endsWith("@domain.com"))
                    insertStmt.setInt(6, 1);
                else
                    insertStmt.setNull(6, java.sql.Types.INTEGER);

                insertStmt.executeUpdate();
                // update new key in bugzilla
                keysRs = insertStmt.getGeneratedKeys();
                keysRs.next();
                int newId = keysRs.getInt(1);
                userMap.put(userId, newId); // put in map for later retrieval
                close(keysRs);

                log.info("Bugzilla user [" + userId + ", " + loginName + " ] > Redmine user [" + newId + ", "
                                + newLoginName + "]");
            }
            log.info(userMap.size() + " users created.");
        } finally {
            close(selectStmt);
            close(insertStmt);
            close(selectRs);
            close(keysRs);
        }
    }

    private void createProjects(Connection destCon, Connection srcCon) throws Exception {
        log.info("Creating project: ABC . . .");
        createProject(destCon, srcCon, "INSERT INTO projects (name, description, identifier, created_on, "
                        + "updated_on) VALUES ('ABC', 'ABC', 'abc', now(), now())", 77);
        // repaet it for more projects

        // Following projects already exist there, just migrate their issues
        log.info("Migrating project: XYZ . . . "
                        + "Redmine project id is 13 for old Bugzilla project id 91");
        createProjectArtifacts(destCon, srcCon, 91, 13);
        projectMap.put(91, 13);
    }

    private void createProject(Connection destCon, Connection srcCon, String insertSql, int bugzillaProjId)
                    throws Exception {
        Statement insertStmt = null;
        ResultSet rs = null;
        try {
            insertStmt = destCon.createStatement();
            insertStmt.executeUpdate(insertSql, new String[] { "id" });
            rs = insertStmt.getGeneratedKeys();
            rs.next();
            int redmineProjId = rs.getInt(1);
            // put in map for later retrieval
            projectMap.put(bugzillaProjId, redmineProjId);
            log.info("   Redmine project id is " + redmineProjId + " for old Bugzilla project id "
                            + bugzillaProjId);

            createProjectArtifacts(destCon, srcCon, bugzillaProjId, redmineProjId);
        } finally {
            close(rs);
            close(insertStmt);
        }
    }

    private void createProjectArtifacts(Connection destCon, Connection srcCon, int bugzillaProjId,
                    int redmineProjId) throws Exception {
        int[] trackers = { 1, 2, 3 };
        Statement insertStmt = null;
        try {
            insertStmt = destCon.createStatement();
            // enable issue tracker module
            insertStmt.executeUpdate("INSERT INTO enabled_modules (project_id, name) VALUES ("
                            + redmineProjId + ", 'issue_tracking')");
            log.info("   Enabled tracker module");

            // create trackers
            log.info("   Creating trackers");
            for (int tracker : trackers)
                insertStmt.executeUpdate("INSERT INTO projects_trackers (project_id, tracker_id) VALUES ("
                                + redmineProjId + ", " + tracker + ")");

            Thread.sleep(1000);
            log.info("   Creating versions");
            createVersions(destCon, srcCon, bugzillaProjId, redmineProjId);

            Thread.sleep(1000);
            log.info("   Assigining project members");
            assignProjectMembers(destCon, srcCon, bugzillaProjId, redmineProjId);

            Thread.sleep(1000);
            log.info("   Creating issue categories");
            createCategories(destCon, srcCon, bugzillaProjId, redmineProjId);

            Thread.sleep(1000);
            log.info("   Migrating issues");
            createIssues(destCon, srcCon, bugzillaProjId, redmineProjId);
        } finally {
            close(insertStmt);
        }
    }

    private void createVersions(Connection destCon, Connection srcCon, int bugzillaProjId, int redmineProjId)
                    throws Exception {
        String SELECT = "SELECT value FROM versions WHERE product_id = " + bugzillaProjId;
        String INSERT = "INSERT INTO versions (project_id, name, created_on, updated_on) VALUES (?, ?, now(), now())";
        Statement selectStmt = null;
        PreparedStatement insertStmt = null;
        ResultSet selectRs = null;
        ResultSet keysRs = null;
        try {
            insertStmt = destCon.prepareStatement(INSERT, new String[] { "id" });
            selectStmt = srcCon.createStatement();
            selectRs = selectStmt.executeQuery(SELECT);
            while (selectRs.next()) {
                String name = selectRs.getString(1);
                log.info("      Creating version: " + name);
                insertStmt.setInt(1, redmineProjId);
                insertStmt.setString(2, name);
                insertStmt.executeUpdate();
                keysRs = insertStmt.getGeneratedKeys();
                keysRs.next();
                int newId = keysRs.getInt(1);
                versionMap.put(bugzillaProjId + name, newId); // store for later retrieval
                keysRs.close();
            }
        } finally {
            close(selectStmt);
            close(insertStmt);
            close(selectRs);
        }
    }

    private void assignProjectMembers(Connection destCon, Connection srcCon, int bugzillaProjId,
                    int redmineProjId) throws Exception {
        String SELECT = "SELECT DISTINCT p.userid, pd.id, p.login_name "
                        + "FROM group_control_map gcm, user_group_map ugm, profiles p, products pd "
                        + "WHERE gcm.group_id = ugm.group_id AND ugm.user_id = p.userid AND gcm.product_id = pd.id "
                        + "AND pd.id = " + bugzillaProjId;
        String INSERT = "INSERT INTO members (user_id, project_id, role_id, created_on) VALUES (?, ?, ?, now())";
        Statement selectStmt = null;
        PreparedStatement insertStmt = null;
        ResultSet selectRs = null;
        try {
            insertStmt = destCon.prepareStatement(INSERT);
            selectStmt = srcCon.createStatement();
            selectRs = selectStmt.executeQuery(SELECT);
            int member = -1;
            while (selectRs.next()) {
                member = selectRs.getInt(1);
                String login_name = selectRs.getString(3);
                log
                                .info("      Assigning member: " + member + "(" + userMap.get(member) + "), "
                                                + login_name);
                insertStmt.setInt(1, userMap.get(member));
                insertStmt.setInt(2, redmineProjId);
                if (login_name.endsWith("@domain.com"))
                    insertStmt.setInt(3, 4); // mark everyone as developer
                else
                    insertStmt.setInt(3, 6); // client
                insertStmt.addBatch();
            }
            insertStmt.executeBatch();
        } finally {
            close(selectRs);
            close(selectStmt);
            close(insertStmt);
        }
    }

    private void createCategories(Connection destCon, Connection srcCon, int bugzillaProjId, int redmineProjId)
                    throws Exception {
        String SELECT = "SELECT id, product_id, name, initialowner FROM components WHERE product_id = "
                        + bugzillaProjId;
        String INSERT = "INSERT INTO issue_categories (project_id, name, assigned_to_id) VALUES (?, ?, ?)";
        Statement selectStmt = null;
        PreparedStatement insertStmt = null;
        ResultSet selectRs = null;
        ResultSet keysRs = null;
        try {
            insertStmt = destCon.prepareStatement(INSERT, new String[] { "id" });
            selectStmt = srcCon.createStatement();
            selectRs = selectStmt.executeQuery(SELECT);
            while (selectRs.next()) {
                insertStmt.setInt(1, redmineProjId);
                String name = selectRs.getString(3);
                name = name.length() <= 30 ? name : name.substring(0, 30);
                log.info("      Creating category: " + name);
                insertStmt.setString(2, name);
                insertStmt.setInt(3, userMap.get(selectRs.getInt(4)));
                insertStmt.executeUpdate();
                keysRs = insertStmt.getGeneratedKeys();
                keysRs.next();
                int newCatId = keysRs.getInt(1);
                close(keysRs);
                categoryMap.put(selectRs.getInt(1), newCatId);
            }
        } finally {
            close(selectRs);
            close(keysRs);
            close(insertStmt);
            close(selectStmt);
        }
    }

    private void createIssues(Connection destCon, Connection srcCon, int bugzillaProjId, int redmineProjId)
                    throws Exception {
        String SELECT = "SELECT bugs.bug_id, bugs.short_desc, longdescs.thetext, bugs.bug_file_loc AS url, bugs.assigned_to, "
                        + "bugs.reporter, bugs.creation_ts, longdescs.bug_when, bugs.priority, bugs.version, bugs.component_id, "
                        + "bugs.bug_status, bugs.resolution, bugs.bug_severity, longdescs.comment_id, longdescs.who, lastdiffed, "
                        + "bugs.cf_phases1, bugs.cf_work "
                        + "FROM bugs, longdescs WHERE bugs.bug_id = longdescs.bug_id AND bugs.product_id = "
                        + bugzillaProjId;
        String INSERT_ISSUE = "INSERT INTO issues (project_id, subject, description, assigned_to_id, author_id, created_on, "
                        + "updated_on, start_date, priority_id, fixed_version_id, category_id, tracker_id, status_id) "
                        + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
        String INSERT_JOURNAL = "INSERT INTO journals (journalized_id, journalized_type, user_id, notes, created_on) "
                        + " VALUES (?, ?, ?, ?, ?)";
        String INSERT_CUSTOM = "INSERT INTO custom_values (customized_type, customized_id, custom_field_id, value) "
                        + "VALUES ('Issue', ?, ?, ?)";
        Statement selectStmt = null;
        PreparedStatement insertIssueStmt = null;
        PreparedStatement insertJournalStmt = null;
        PreparedStatement insertCustomStmt = null;
        ResultSet selectRs = null;
        ResultSet keysRs = null;
        try {
            insertIssueStmt = destCon.prepareStatement(INSERT_ISSUE, new String[] { "id" });
            insertJournalStmt = destCon.prepareStatement(INSERT_JOURNAL);
            insertCustomStmt = destCon.prepareStatement(INSERT_CUSTOM);
            selectStmt = srcCon.createStatement();
            selectRs = selectStmt.executeQuery(SELECT);
            int bugId = -1;
            int redmineBugId = -1;
            while (selectRs.next()) {
                int currentBugId = selectRs.getInt(1);
                // first record for the bug
                if (currentBugId != bugId) {
                    bugId = currentBugId;
                    log.info("    " + bugId + " >>>");
                    insertIssueStmt.setInt(1, redmineProjId);
                    insertIssueStmt.setString(2, selectRs.getString(2)); // subject
                    // description along with URL
                    insertIssueStmt.setString(3, selectRs.getString(3) + "\n\n" + selectRs.getString(4));
                    insertIssueStmt.setInt(4, userMap.get(selectRs.getInt(5))); // assigned_to
                    insertIssueStmt.setInt(5, userMap.get(selectRs.getInt(6)));// reporter
                    insertIssueStmt.setTimestamp(6, selectRs.getTimestamp(7)); // created_on
                    insertIssueStmt.setTimestamp(7, selectRs.getTimestamp(17)); // updated_on
                    insertIssueStmt.setDate(8, selectRs.getDate(7)); // start_date
                    insertIssueStmt.setInt(9, priority.get(selectRs.getString(9)));// priority
                    // fixed_version_id - do post processing to set 0 values to null/empty
                    insertIssueStmt.setInt(10, versionMap.get(bugzillaProjId + selectRs.getString(10)));
                    insertIssueStmt.setInt(11, categoryMap.get(selectRs.getInt(11))); // component/category
                    insertIssueStmt.setInt(12, tracker.get(selectRs.getString(14))); // tracker
                    insertIssueStmt.setInt(13, status.get(selectRs.getString(12))); // status
                    // (resolution)
                    insertIssueStmt.executeUpdate();
                    keysRs = insertIssueStmt.getGeneratedKeys();
                    keysRs.next();
                    redmineBugId = keysRs.getInt(1);
                    close(keysRs);
                    issueMap.put(bugId, redmineBugId);

                    // Custom values Severity(5), Defect Injection(3) and Defect Cause(4)
                    insertCustomStmt.setInt(1, redmineBugId);
                    insertCustomStmt.setInt(2, 5); // severity
                    insertCustomStmt.setString(3, selectRs.getString(14));
                    insertCustomStmt.addBatch();
                    insertCustomStmt.setInt(1, redmineBugId);
                    insertCustomStmt.setInt(2, 3); // injection
                    insertCustomStmt.setString(3, selectRs.getString(18));
                    insertCustomStmt.addBatch();
                    insertCustomStmt.setInt(1, redmineBugId);
                    insertCustomStmt.setInt(2, 4); // cause
                    insertCustomStmt.setString(3, selectRs.getString(19));
                    insertCustomStmt.addBatch();

                    // Map Bugzilla CC to Redmine Watchers
                    createWatchers(destCon, srcCon, bugId, redmineBugId);
                    // log.info("      Watchers (CC list) migrated");

                    // process attachments (11931 has attachments)
                    handleAttachments(destCon, srcCon, bugId, redmineBugId);
                    // log.info("      Attachments migrated");
                } else {
                    // Create journal entries (comments on issues)
                    insertJournalStmt.setInt(1, redmineBugId);
                    insertJournalStmt.setString(2, "Issue");
                    insertJournalStmt.setInt(3, userMap.get(selectRs.getInt(16)));
                    insertJournalStmt.setString(4, selectRs.getString(3));
                    insertJournalStmt.setTimestamp(5, selectRs.getTimestamp(8));
                    insertJournalStmt.addBatch();
                }
            }
            insertJournalStmt.executeBatch();
            log.info("      Created Journal entries");
            insertCustomStmt.executeBatch();
            log.info("      Created severity, cuase and injection");

            // Map Bugzilla Dependencies to Redmine Relations and Duplicates
            for (Integer issue : issueMap.keySet()) {
                updateIssueRelations(destCon, srcCon, issue);
                // log.info("      Updated related and duplicate issues");
            }
        } finally {
            close(selectRs);
            close(selectStmt);
            close(insertIssueStmt);
            close(insertJournalStmt);
            close(insertCustomStmt);
        }
    }

    private void createWatchers(Connection destCon, Connection srcCon, int bugzillaBugId, int redmineBugId)
                    throws Exception {
        String INSERT = "INSERT INTO watchers (watchable_id, user_id, watchable_type) VALUES (?, ?, 'Issue')";
        String SELECT = "SELECT bug_id, who FROM cc where bug_id = " + bugzillaBugId;
        Statement selectStmt = null;
        PreparedStatement insertStmt = null;
        ResultSet selectRs = null;
        String cc = null;
        try {
            insertStmt = destCon.prepareStatement(INSERT);
            selectStmt = srcCon.createStatement();
            selectRs = selectStmt.executeQuery(SELECT);
            int count = 0;
            while (selectRs.next()) {
                insertStmt.setInt(1, redmineBugId);
                int who = selectRs.getInt(2);
                if (count == 0)
                    cc = "      Adding as watcher: " + who + "(" + userMap.get(who) + ")";
                else
                    cc += ", " + who + "(" + userMap.get(who) + ")";
                insertStmt.setInt(2, userMap.get(who));
                insertStmt.addBatch();
                count++;
            }
            insertStmt.executeBatch();
        } finally {
            close(selectRs);
            close(selectStmt);
            close(insertStmt);
            if (cc != null && cc.length() > 0)
                log.info(cc);
        }
    }

    private void handleAttachments(Connection destCon, Connection srcCon, int bugzillaBugId, int redmineBugId)
                    throws Exception {
        String SELECT = "SELECT a.filename, a.creation_ts, ad.thedata, a.attach_id, a.bug_id, a.mimetype, a.submitter_id, a.description "
                        + "FROM attachments a, attach_data ad WHERE a.attach_id = ad.id and a.bug_id = "
                        + bugzillaBugId;
        String INSERT = "INSERT INTO attachments (container_id, container_type, filename, disk_filename, filesize, "
                        + "content_type, digest, downloads, author_id, created_on, description) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
        Statement selectStmt = null;
        PreparedStatement insertStmt = null;
        ResultSet selectRs = null;
        try {
            insertStmt = destCon.prepareStatement(INSERT);
            selectStmt = srcCon.createStatement();
            selectRs = selectStmt.executeQuery(SELECT);
            while (selectRs.next()) {
                String fileName = selectRs.getString(1);
                Timestamp fileTs = selectRs.getTimestamp(2);
                // disk file name in redmine is yymmddhhmmss_filename.ext
                String diskFileName = sdf.format(fileTs) + "_" + fileName;
                log.info("      Saving attachment: " + fileName + "(" + diskFileName + ") for issue "
                                + bugzillaBugId + "(" + redmineBugId + ")");
                Blob blob = selectRs.getBlob(3);
                byte[] allBytesInBlob = blob.getBytes(1, (int) blob.length());
                File file = new File(filePath, diskFileName);
                file.createNewFile();
                FileOutputStream fos = new FileOutputStream(file);
                fos.write(allBytesInBlob);
                fos.flush();
                fos.close();

                insertStmt.setInt(1, redmineBugId);
                insertStmt.setString(2, "Issue");
                insertStmt.setString(3, fileName);
                insertStmt.setString(4, diskFileName);
                insertStmt.setInt(5, allBytesInBlob.length);
                insertStmt.setString(6, selectRs.getString(6)); // mime type
                insertStmt.setString(7, ""); // digest- leave it blank
                insertStmt.setInt(8, 0); // downloads
                insertStmt.setInt(9, userMap.get(selectRs.getInt(7))); // author
                insertStmt.setTimestamp(10, fileTs);
                insertStmt.setString(11, selectRs.getString(8));
                insertStmt.addBatch();
            }
            insertStmt.executeBatch();
        } finally {
            close(selectRs);
            close(selectStmt);
            close(insertStmt);
        }
    }

    private void updateIssueRelations(Connection destCon, Connection srcCon, int bugzillaBugId)
                    throws Exception {
        String SELECT_DUP = "SELECT dupe_of, dupe FROM duplicates WHERE dupe_of = " + bugzillaBugId;
        String SELECT_DEP = "SELECT blocked, dependson FROM dependencies WHERE blocked = " + bugzillaBugId;
        String INSERT = "INSERT INTO issue_relations (issue_from_id, issue_to_id, relation_type) VALUES (?, ?, ?)";
        Statement selectStmt = null;
        PreparedStatement insertStmt = null;
        ResultSet rs = null;
        try {
            insertStmt = destCon.prepareStatement(INSERT);
            selectStmt = srcCon.createStatement();
            rs = selectStmt.executeQuery(SELECT_DUP);
            while (rs.next()) {
                insertStmt.setInt(1, issueMap.get(rs.getInt(1)));
                insertStmt.setInt(2, issueMap.get(rs.getInt(2)));
                insertStmt.setString(3, "duplicates");
                insertStmt.addBatch();
            }
            close(rs);
            rs = selectStmt.executeQuery(SELECT_DEP);
            while (rs.next()) {
                insertStmt.setInt(1, rs.getInt(1));
                insertStmt.setInt(2, rs.getInt(2));
                insertStmt.setString(3, "blocks");
                insertStmt.addBatch();
            }
            insertStmt.executeBatch();
        } finally {
            close(rs);
            close(selectStmt);
            close(insertStmt);
        }
    }

    private void close(ResultSet rs) {
        if (rs == null)
            return;
        try {
            rs.close();
        } catch (Exception e) {
            log.error("Failed to close ResultSet", e);
        }
    }

    private void close(Statement stmt) {
        if (stmt == null)
            return;
        try {
            stmt.close();
        } catch (Exception e) {
            log.error("Failed to close PreparedStatement", e);
        }
    }

}