Monday 7 October 2013

[Book Review] ElasticSearch Server for fast scalable flexible search solution

ElasticSearch Server 



This is a good book for someone who wants understand and try out the latest search framework ElasticSearch built on top of Lucene. This claims to provide the best, flexible, distributed and enterprise Search Framework

It covers almost all the topic that is required
  1. Introduction to Elastic Search
  2. Understanding Shards, Nodes and cluster.
  3. Indexing and Searching Techniques
  4. Templates, Mapping
  5. Different API's
  6. Plugins.
  7. etc.,

I would suggest this book for someone interested in ElasticSearch Server.

Thursday 26 September 2013

Java: Code to unquote string using Regex


str="\"hello\"" // or "'hello'"

public static unquote(String str)
{
        str = str.trim();
if (str.startsWith("'") && str.endsWith("'"))
{
    str = match(str, "(?<=').*(?=')");
}
else if (str.startsWith("\"") && str.endsWith("\""))
{
         str = match(str, "(?<=\").*(?=\")");
}
}

//matches the pattern
private static String match(final String string, final String regex)
{
final Pattern pattern = Pattern.compile(regex);
final Matcher matcher = pattern.matcher(string);
matcher.find();
return matcher.group(); 
}

[Shared Webinar] Getting Started with ElasticSearch

Click on the Below Image and fill in details to watch the Webinar


Saturday 21 September 2013

Quick Notes : Practical usage of collections

Different Types of Collections and when to Use them

Set

HashSet : 

  • When you want unique set to objects. 
  • Make sure equals and hashcode is implemented.
  • This does not maintain the order of object addition. As the name says it is Hashed

TreeSet:
  • You can have the list ordered the way you want, by passing in a comparator.
  • Example: new TreeSet(String.CASE_INSENSITIVE_ORDER)
  • It automatically orders the objects that you add accordingly

LinkedHashSet
  • This gives us the flexibility of HashSet (uniqueness) and 
  • also maintains the insertion-order of objects

Map

HashMap :
  • Easier way of holding Key Value Pair
  • Does not maintain the insertion-order (Hashed)
TreeMap:
  • Sorted Map, Sorting of keys determined by the comparator in the constructor
  • Quickly create a map of case insensitive keys as follows
    new TreeMap(String.CASE_INSENSITIVE_ORDER)
  • Throws a NullPointerException if you use get without a null check.
    treemap.get(null), so better add a null check before invoking get()
LinkedHashMap:
  • Extends HashMap. But, preserves of the insertion order

List:

ArrayList:
  • Quicker way of maintaining a list of objects
  • can add duplicate values.
  • It is advisable to use this collection if we expect more number of gets() than add() or set()
LinkedList:
  • Performs better when you have more additions and insertions than searching the list

Tuesday 13 August 2013

Deleting Public Keys from GPG and RPM key rings



In order to delete the public keys from the key ring, we would require the key_id. The key_id can be idenfied using the below steps using the file where the public key is stored.

To get the Key Details

gpg --throw-keyids < [Key-File-Absolute Path]

pub  1021D/1689B887 2005-03-11 ABC Company 
sub  1048f/47E5X96D 2005-03-11  [expires: 2015-03-09]

To get the Key-ID from the keyfile

key_id=${gpg --throw-keyids < [Key-File-Absolute Path] | head -1 |  awk {'print $2'} |  cut -d '/' -f2 |  tr [A-Z] [a-z]}

1689B887

Delete from GPG keyring

gpg --delete-key  --yes $key_id

Delete from RPM keyring

rpm -e --allmatches gpg-pubkey-$key_id

Wednesday 17 July 2013

Linux script to display tree structure of directory

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'


Output from  a folder. This displays "ONLY" directories
 .
   |-resources
   |-src
   |---java
   |-target


(or)


yum install tree

tree -A
.
├── resources
├── src
│   ├── Hello.java
│   └── java
└── target

-A to be used if you use garbage chars as follows

.
âââ resources
âââ src
â   âââ Hello.java
â   âââ java
âââ target



Reference
http://ssteiner.wordpress.com/2009/05/12/quick-linux-tree-view-of-directory-contents/



Sunday 7 July 2013

Regex to find non-quoted string match

Created a below regex to  identify , that is not quoted. 

(,) (?=(?:[^\"]|\"[^\"]*'\")*$),

(,) 
    (?=   
           (?:                               # non capturing group
                 [^\"]|                     # string not quoted 
                 \"[^\"]*'\"           # string inside quotes
            )*$                            # many a times until end of string
     )


This has a an issue when you really run it with a lengthy string
positive lookahead  + alteration +  * + lengthy string  +  heap size crunch.




So the below regex will fetch you the 

(,)(?=(?:[^"]*"[^"]*")*[^"]*$)

(,)
   (?=                            # look ahead
       (?:                         # non capturing group, ignore string within group 
           [^"]*                 # group of non-quoted text  
           "[^"]*"               # followed by quoted quote + non quoted (*)+ quote  
        )*                        # this group can be 0 or many
    [^"]*                       # capturing group    (non quoted)
   $)                            # till end of string



Regex Tester Online: http://regexpal.com/


String used to Test 
(state "locked" OR state" not active" NOT state"pending or " OR state"running" AND state" and completed" OR state"error" OR state"warning") AND NOT hidden"true" AND NOT type"Background" AND NOT cycle"true" AND parent"null" AND test"asdfasdf asdfhasd fha sdfhas udfhuasdhfu asudfhasudfhuas fhuihas udfhuashd ufhasuidfhua shdfuas hduhuhasd fuhasdufhas duhuhu hasufhuasdhfdfsdfasdfsdasdsad7777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777aasda777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777sd7777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777as7777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777d7end")





Wednesday 12 June 2013

Map with Case-Insensitive Keys

Below is the easy way, for case-insensitive key searches in a Map.


Map<String, String[]> treemap = new TreeMap<String, String[]>(String.CASE_INSENSITIVE_ORDER);

treemap.put("name1","value1");
treemap.put("NAME2","value2");
 
treemap.get("NAME1")   //you get "value1"
treemap.get("naME2")   //you get "value2"



[WARNING]  treemap.get(null) throws NulllPointerException in certain versions of Java

Tuesday 16 April 2013

Hibernate supporting case-insensitive search with Postgres citext

There are issues with  Hibernate supporting case-insensitive search on citext, with Hibernate Criteria’s and Bind Parameters

Problem :

Assume this is the table structure with the data

CREATE TABLE parententity
(
  id character varying(255) NOT NULL,
  firstname citext,
  lastname character varying(255),
  CONSTRAINT parententity_pkey PRIMARY KEY (id)
)

                  id                                                           |   firstname  |  lastname
--------------------------------------+-----------+------------
AB83B745-BDCE-4033-ADF9-4C8FBC2071A5 |  vidhya         |  shanker
D753F492-F9BA-417E-ABA9-3CE50D90D65A  |  Srividhya    |  Umashanker

Here are our queries

Same Case value
Query q = dao.getSession().createSQLQuery("select * from ParentEntity where firstName=:name")
              .addEntity(ParentEntity.class)
              .setString("name", "Srividhya");
      List l = q.list();
Hibernate fetches  AB83B745-BDCE-4033-ADF9-4C8FBC2071A5 | Srividhya | Umashanker

Where as

Different value
Query q = dao.getSession().createSQLQuery("select * from ParentEntity where firstName=:name")
              .addEntity(ParentEntity.class)
              .setString("name", "SRIVIDHYA");
      List l = q.list();

Hibernate does not fetch any records

which is expected to do a case insensitive fetch,  does not seem to fetch any record. Internally it converts to varchar and does a case sensitive check.
Looks like this is a known  as per Postgres JDBC implementation

package org.postgresql.jdbc2;
    public abstract class AbstractJdbc2Statement implements BaseStatement
   {
       
protected AbstractJdbc2Connection(String host, int port, String user, String database, Properties info, String url) throws SQLException
     {
          String stringType = info.getProperty("stringtype");
        if (stringType != null) {
            if (stringType.equalsIgnoreCase("unspecified"))
                bindStringAsVarchar = false;
            else if (stringType.equalsIgnoreCase("varchar"))
                bindStringAsVarchar = true;
            else
                throw new PSQLException(GT.tr("Unsupported value for stringtype parameter: {0}", stringType),
                PSQLState.INVALID_PARAMETER_VALUE);
        }
     }
    
    public void setString(int parameterIndex, String x) throws SQLException
    {
….
setString(parameterIndex, x, (connection.getStringVarcharFlag() ? Oid.VARCHAR : Oid.UNSPECIFIED));
    }



SOLUTION:


There also seems to be an option now in the postgres JDBC documentation, that allows you to add a connection option stringtype. The option is set in the connection url as
jdbc:postgresql://localhost/test?stringtype=unspecified. The postgres documentation says…

stringtype = String
Specify the type to use when binding PreparedStatement parameters set via setString(). If stringtype is set to varchar (the default), such parameters will be sent to the server as varchar parameters. If stringtype is set to
unspecified, parameters will be sent to the server as untyped values, and the server will attempt to infer an appropriate type. This is useful if you have an existing application that uses setString() to set parameters that
are actually some other type, such as integers, and you are unable to change the application to use an appropriate method such as setInt().


When tried as follows in hibernate.properties  the functionality seems to work both for citext, varchar and text

hibernate.connection.url=jdbc:postgresql://192.168.1.1:5432/test?stringtype=unspecified

Thursday 11 April 2013

Hibernate SQLRestriction using alias of the join Table.


Problem:

We Faced an issue which using SQLRestricion in the subcriteria and we use an table aliasCriteria c1 = dao.getSession().createCriteria(Parent.class,"p")

      .createCriteria("p.child", "c", Criteria.LEFT_JOIN)
      .add(Restrictions.sqlRestriction("c.address ~* 'Address%'"));

print(dao, c1);
The SQL that gets generated is as follows and failed

select this_.id as id3_0_, this_.address as address3_0_, this_.phone as phone3_0_, this_.firstName as firstName3_0_
from
   Parent this_
   left outer join Child this_1_
on
   this_.id=this_1_.id
where
   c.address ilike 'Address%'

Expected sql that uses the alias in my where clause.

select this_.id as id3_0_, this_.address as address3_0_, this_.phone as phone3_0_, this_.firstName as firstName3_0_
from
   Parent this_
   left outer join Child c
on
   this_.id=c.id
where
   c.address ilike 'Address%'


Root Cause:


We identified that the problem was with SQLRestriction implementation in hibernate, which does not care about Alias names. So we took an approach, to not use SQLRestriction, instead extended SimpleExpression to accomodate Regex 



Solution:


Instead of using SQLRestriction, we resolved it by extending SimpleExpression, which solved the problem

public class CustomSimpleExpression extends SimpleExpression{
    public CustomSimpleExpression(String propertyName, Object value, String op){
       super(propertyName, value, op);
    }
}


public class CustomRestrictions{
     public static SimpleExpression regex(String propertyName, Object value){
         return new CustomSimpleExpression(propertyName, value, " ~* ");
      }
}


Criteria c1 = dao.getSession().createCriteria(Parent.class,"p")
      .createCriteria("p.child", "c", Criteria.LEFT_JOIN)
      .add(CustomRestrictions.regex(c.address,"Address%"));

print(dao, c1);

[Shared Presentation] Introduction to NoSQL by Martin Fowler

I came across this wonderful presentation by Martin Fowler on NOSQL Databases.
Anyone who wants to know the basics. Please find the time to watch this

Monday 11 February 2013

SQL Joins - Quick examples






SQL joins are used to query data from two or more tables, based on a relationship between certain columns in these tables.


SQL JOIN

The JOIN keyword is used in an SQL statement to query data from two or more tables, based on a relationship between certain columns in these tables.

Tables in a database are often related to each other with keys.


A primary key is a column (or a combination of columns) with a unique value for each row. Each primary key value must be unique within the table. The purpose is to bind data together, across tables, without repeating all of the data in every table.



Different SQL JOINs

Before we continue with examples, we will list the types of JOIN you can use, and the differences between them.
INNER JOIN:Return rows when there is at least one match in both tables
OUTER LEFT JOIN:Return all rows from the left table, even if there are no matches in the right table
OUTER RIGHT JOIN:Return all rows from the right table, even if there are no matches in the left table
FULL JOIN:Return rows when there is a match in one of the tables


Tables used for example:

CREATE TABLE cities (

    name character varying(20),
    location character varying(10)
);


CREATE TABLE weather (
    city character varying(20),
    temp_lo integer,
    temp_hi integer
);

Inner join

The INNER JOIN keyword returns rows when there is at least one match in both tables. If there are rows in "Cities" that do not have matches in "Weather", those rows will NOT be listed and vice versa.

IN the below example the "Delhi" and "Chennai" are the ones that are in both Cities and weather

SELECT * FROM cities INNER JOIN weather ON (cities.name=weather.city);




Left outer join

This query is called a left outer join because the table mentioned on the left of the join operator will have each of its rows in the output at least once, whereas the table on the right will only have those rows output that match some row of the left table. When outputting a left-table row for which there is no right-table match, empty (null) values are substituted for the right-table columns.

In the below example, "Calcutta "/"Delhi"/"Chennai" from cities are the selected rows, eventhough weather does not have "Calcutta"

SELECT * FROM cities LEFT OUTER JOIN weather ON (cities.name=weather.city);



Right outer join

This query is called a right outer join because the table mentioned on the right of the join operator will have each of its rows in the output at least once, whereas the table on the left will only have those rows output that match some row of the right table. When outputting a right-table row for which there is no left-table match, empty (null) values are substituted for the left-table columns.

In the Below Example, "Delhi"/"Chennai"/"Bangalore" from weather are the selected rows eventhough cites does not contain "Bangalore".


SELECT * FROM cities RIGHT OUTER JOIN weather ON (cities.name=weather.city);


Full Join

The FULL JOIN keyword return rows when there is a match in one of the tables.

IN the Below Example "Delhi"/"Chennai"/"Calcutta"/"Bangalore" all rows from cities and weather are listed.

SELECT * FROM cities FULL JOIN weather ON (cities.name=weather.city);




Self join

We can also join a table against itself. This is called a self join. 

As an example, suppose we wish to find all the weather records that are in the temperature range of other weather records. So we need to compare the temp_lo and temp_hi columns of each weather row to the temp_lo and temp_hi columns of all other weather rows. We can do this with the following query:
SELECT W1.city, W1.temp_lo AS low, W1.temp_hi AS high, 
W2.city, W2.temp_lo AS low, W2.temp_hi AS high 
FROM weather W1, weather W2WHERE W1.temp_lo < W2.temp_lo
AND W1.temp_hi > W2.temp_hi;





References

Thursday 7 February 2013

Setup remote server profiling with Jprofiler7 (Windows to Linux)

What is Profiling?

Analyze the performance of a Java program/JVM, by monitoring and collecting runtime data on object allocations, garbage collection cycles, CPU and memory usage, object references, method time stamps, threads and object interactions.


What is JProfiler?


Wiki says

"JProfiler is a commercially licensed Java profiling tool developed by ej-technologies GmbH, targeted at Java EE and Java SE applications."


At all times this is my first choice when it comes to profiling.


How to setup Remote profiling with Jprofiler  from Windows to Linux?

Summary 

  1. Download the Jprofiler EXE for windows and  tar.gz for linux
  2. Copy the tar file to linux and untar
  3. Remove linux firewall and selinux restrictions (if applicable)
  4. Run the JProfiler agent in linux
  5. Install JProfiler on windows and set the trial/commerical licence
  6. From JProfiler GUI, connect to the agent running on linux(Step 4)
  7. Start profiling

Lets walk through each step and see it working!


1. Download the Jprofiler EXE for windows and tar.gz for linux from here


2. Copy the tar file to linux and untar

  • Copy the file to a folder in linux

  • untar it using  "tar -xvf <tar filename>"
    This will untar the contents to a "jprofiler7" folder.


3. Remove linux firewall and selinux restrictions (if applicable)

Linux firewall and selinux might be turned on. Just disable those using the below commands. We can be enabled later after profiling
  • turnoff firewall :  service iptables stop
  • turnoff selinux :  setenforce 0


4. Run the JProfiler agent in linux

  • We should be linking the "server to be profiled" with the "untar'd jprofiler agent"
  • To do this we should add -agent VM parameter while running the server.
  • The same port specified here should be used when connecting from the GUI.
-agentpath:<installed folder> /jprofiler7/bin/linux-x64/libjprofilerti.so=port=<port number>

example :  -agentpath:/root/jprofiler7/bin/linux-x64/libjprofilerti.so=port=11002


Sample command to startup a jetty server

java -Djetty.home=/jetty -agentpath:/root/jprofiler7/bin/linux-x64/libjprofilerti.so=port=11002 -jar /jetty/start.jar

Now the server will startup with jprofiler agent, listening to Jprofiler GUI connections at port 11002.

5. Install JProfiler on windows and set the trial/commerical licence

Using the Executable we downloaded in step 1, install Jprofiler and set the appropriate licence and then start the Jprofiler

6. From JProfiler GUI, connect to the agent running on linux machine(Step 4)

  • Follow the steps as per the below diagram, the order are specified within the yellow circle.
  • Set the Linux Host IP or name and the same port specified with -agent in step 4. Example 11002.

You will pass through these screenshots as below:





7. Start Profiling

If you see the below screen, then it means you are connected.



References:



Tuesday 5 February 2013

Different ways to Skip Maven Tests

 2 Different ways to skip tests with Maven.



  • mvn –o clean install  -DskipTests  - Use this to just avoid “RUNNING” the tests
  • mvn –o clean install –Dmaven.test.skip=true – Use this to avoid “BOTH COMPILE  & RUNNING” the tests.  (skip completely when tests have compilation errors)
 
  These 2 may appear to be the same, but they perform differently.