Apache Shiro Part 1 – Basics

Apache Shiro, originally called JSecurity, is Java security framework. It was accepted and became Apache top level project in 2010. It aims to be powerful and easy to be used.

The project is in active development with active both users and developers mailing lists. Most important areas are documented on its web page. However, it has lot of gaps in documentation. It is not possible to learn to use most Shiro features from documentation alone. Luckily, the code is well commented and where I tried it was also easily readable.

Main Shiro features are:

  • authentication,
  • authorization,
  • cryptography,
  • session management.

In this article article we try and demonstrate various Shiro features. We start with simple unsecured web application, then we add security features into it. All code code is available in
SimpleShiroSecuredApplication project on Github.





Unsecured Application

Unsecured application code is located in
unsecured_application branch. Application represents an internal system for a fictional company. The company has four departments:

  • administrators,
  • repairmen,
  • scientists,
  • sales.

Each department has its own page. Each page contains buttons that are used by users to do their work. When user presses button, the work is done. For example, any repairmen can go to repairmen page and press button "Repair Refrigerator". The button repairs refrigerator and shows success message.

Each user have his own account page. Account page contains user’s private data. Since unsecured application has no users yet, account page does nothing. Additionally, there is a page which contains all application functions. Everything anybody can do is possible to be done on this page.

Anybody can do any work and see all the pages.Sample application is run in test class
RunWaitTest. It is not the best practice to use unit test this way, but it is not important now. If you run the class the application will be available at http://localhost:9180/simpleshirosecuredapplication/ url.

Adding Authentication

First, we have to verify user’s identity. The easiest and most standard authentication is done through user name and password. User fills in his user name and password and system verifies whether supplied values match with some user account.

For simplest applications, it is sufficient to store user names and passwords in plain text files. In more realistic scenario, user name and password are stored in persistent storage or the verification is done through other system such as ldap or active directory. Shiro supports all mentioned authentication methods. If out of the box authentication features are not sufficient, it is possible to extend the framework with own verification implementation.

In this chapter, we add user name and password based authentication into the application. User name and password are stored in static plain-text Shiro ini file.

New requirements: It is possible to log in and log out users. Application is to be accessible only for logged users. Successful log in redirects user into his own account page. All application functions and pages are still accessible to any logged user.

Needed steps:

  • add Apache Shiro,
  • create log in page,
  • configure users and password,
  • create log out page.


Add Apache Shiro

Shiro is integrated into web application through servlet filters. A filter intercepts requests and responses before servlet and performs all necessary tasks (such as identifying currently logged user, attaching logged user to current thread, … ). Default Shiro filters provide basic security features such as:

  • enforcing user log in,
  • enforcing ssl,
  • checking of page access rights.

If you want to learn more about default Shiro filters, good place to start is
DefaultFilter enumeration. It lists all Shiro filters available by default. If those are not sufficient for your needs, you may create custom one.

We will use highly configurable
IniShiroFilter. It reads Shiro configuration from ini file and initializes security framework. It does not perform any security checks. Permission checks, user login, protocol checking etc. are all delegated to either default or custom filters. IniShiroFilter only initialize them.

Ini configuration is described in both
documentation and
javadoc. Ini file configuration has four sections:

  • Section [main] contains Shiro initialization. Filters and custom objects are configured here.
  • Section [users] defines users, passwords and roles.
  • Section [roles] associates roles with permissions.
  • Section [urls] specifies access rights to application pages (urls). It is done by binding either default or custom filters to urls.

Add Apache Shiro dependency to pom.xml:

<properties>
    <shiro.version>1.1.0</shiro.version>
</properties>
<dependencies>
    <dependency>
        <groupid>org.apache.shiro</groupid>
        <artifactid>shiro-core</artifactid>
        <version>${shiro.version}</version>
    </dependency>
    <dependency>
        <groupid>org.apache.shiro</groupid>
        <artifactid>shiro-web</artifactid>
        <version>${shiro.version}</version>
    </dependency>
</dependencies>

Create Shiro.ini file and put it on classpath. Configure web.xml to call IniShiroFilter before each request:

<filter>
    <filter-name>ShiroFilter</filter-name>
    <filter-class>org.apache.shiro.web.servlet.IniShiroFilter</filter-class>
    <init-param>
        <param-name>configPath</param-name>
        <param-value>classpath:Shiro.ini</param-value>
    </init-param>
</filter>

<filter-mapping>
    <filter-name>ShiroFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>


Create Login Page

Login page is simple html page with submit button, user name and password fields. Login functionality is handled by default Shiro authc filter. Authc filter allows url access only to logged in users. If the user is not logged in, filter will redirect him to login page.

Form on login page must have name ‘loginform’ and its submit method must be ‘post’. Create
login.jsp page:

<form name="loginform" action="" method="post">
<table align="left" border="0" cellspacing="0" cellpadding="3">
    <tr>
        <td>Username:</td>
        <td><input type="text" name="user" maxlength="30"></td>
    </tr>
    <tr>
        <td>Password:</td>
        <td><input type="password" name="pass" maxlength="30"></td>
    </tr>
    <tr>
        <td colspan="2" align="left"><input type="checkbox" name="remember"><font size="2">Remember Me</font></td>
    </tr>
    <tr>
        <td colspan="2" align="right"><input type="submit" name="submit" value="Login"></td>
    </tr>
</table> 
</form>

Enable authc filter for all application pages:

[main] 
# specify login page
authc.loginUrl = /simpleshirosecuredapplication/account/login.jsp

# name of request parameter with username; if not present filter assumes 'username'
authc.usernameParam = user
# name of request parameter with password; if not present filter assumes 'password'
authc.passwordParam = pass
# does the user wish to be remembered?; if not present filter assumes 'rememberMe'
authc.rememberMeParam = remember

# redirect after successful login
authc.successUrl  = /simpleshirosecuredapplication/account/personalaccountpage.jsp

[urls]
# enable authc filter for all application pages
/simpleshirosecuredapplication/**=authc

Update: Shiro automatically performs context-relative path matching. As SimpleShiroSecuredApplication does not have context path set, full paths in Shiro.ini are necessary. However, if application context path would be /simpleshirosecuredapplication, then paths could be relative: e.g. simple /**=authc or /account/personalaccountpage.jsp.

As it is not safe to send non-encrypted user name and password through network, we should force ssl logins. Ssl filter does exactly that. It has an optional parameter: ssl port number. If the port parameter is omitted, it uses default ssl port 443.

Before configuring ssl in Shiro, we have to enable it on web server. How to do that depends on web server. We show how to enable it in Jetty. First, create keystore with self signed certificate:

keytool -genkey -keyalg RSA -alias jetty -keystore keystore -storepass secret -validity 360 -keysize 2048

Answer all questions and in the end press enter so the keystore password and key passwords are the same.

Second, add the keystore to the project and configure Jetty to use ssl. Java code is available in
AbstractContainerTest class.

Now, it is possible to configure ssl filter in Shiro.ini:

[urls]
# force ssl for login page
/simpleshirosecuredapplication/account/login.jsp=ssl[8443],authc
# enable authc filter for the all application pages; as Shiro reads urls from up to down, must be last
/simpleshirosecuredapplication/**=authc


Configure Users and Password

SimpleShiroSecuredApplication is now available only for logged users. We now need to add some users so people can log in. Configuration is done in [users] section of Shiro.ini file. Format of section entries is:

username = password, roleName1, roleName2, ..., roleNameN

Following section creates seven users, all have the same password ‘heslo’:

[users]
administrator=heslo,Administrator
friendlyrepairmen=heslo,repairmen
unfriendlyrepairmen=heslo,repairmen
mathematician=heslo,scientist
physicien=heslo,scientist
productsales=heslo,sales
servicessales=heslo,sales

It is now possible to log in into the application. However, no reasonable error message is shown if the user makes a mistake. Moreover, passwords are stored in plain text file.


Error Handling

If user makes an error while logging in, Shiro redirects him back to the login page. The page looks exactly the same as before, which may confuse the user.

New requirement: show error message after each unsuccessful log in attempt.

Any time authentication error occurs, an exception is thrown. By default, form authentication filter catches the exception and stores its class name in request parameter. As we wish to customize data send to page, we have to extend
FormAuthenticationFilter and override
setFailureAttribute method:

@Override
protected void setFailureAttribute(ServletRequest request, AuthenticationException ae) {
  String message = ae.getMessage();
  request.setAttribute(getFailureKeyAttribute(), message);
}

Replace form authorization filter with
VerboseFormAuthenticationFilter and configure it to use ‘simpleShiroApplicationLoginFailure’ request attribute to hold error information:

[main]
# replace form authentication filter with verbose filter 
authc = org.meri.simpleshirosecuredapplication.servlet.VerboseFormAuthenticationFilter
# request parameter with login error information; if not present filter assumes 'shiroLoginFailure'
authc.failureKeyAttribute=simpleShiroApplicationLoginFailure

Show error in login.jsp page:

<% 
  String errorDescription = (String) request.getAttribute("simpleShiroApplicationLoginFailure");
  if (errorDescription!=null) {
%>
Login attempt was unsuccessful: <%=errorDescription%>
<% 
  }
%>

Beware: real application should not show too much login error information. Message ‘Login attempt was unsuccessful.’ with no further info is usually enough.

Hashing Passwords

Current application version has all passwords stored in plain text. It is better to store and compare only password hashes.

Objects responsible for authentication are called
realms. By default, Shiro uses
IniRealm with pluggable password matcher to compare passwords. We will replace passwords in ini by their SHA-256 hashes and configure IniRealm to use SHA-256 hashing matcher.

Generate SHA-256 hash of password:

import org.apache.shiro.crypto.hash.Sha256Hash;

public static void main(String[] args) {
    Sha256Hash sha256Hash = new Sha256Hash("heslo");
    System.out.println(sha256Hash.toHex());
}

Configure Shiro to compare password hashes instead of password itself:

[main] 
# define matcher matching hashes instead of passwords
sha256Matcher = org.apache.shiro.authc.credential.HashedCredentialsMatcher
sha256Matcher.hashAlgorithmName=SHA-256

# enable matcher in iniRealm (object responsible for authentication)
iniRealm.credentialsMatcher = $sha256Matcher

Replace users passwords by password hashes:

[users]
administrator=56b1db8133d9eb398aabd376f07bf8ab5fc584ea0b8bd6a1770200cb613ca005, Administrator
friendlyrepairmen=56b1db8133d9eb398aabd376f07bf8ab5fc584ea0b8bd6a1770200cb613ca005, repairmen
unfriendlyrepairmen=56b1db8133d9eb398aabd376f07bf8ab5fc584ea0b8bd6a1770200cb613ca005, repairmen
mathematician=56b1db8133d9eb398aabd376f07bf8ab5fc584ea0b8bd6a1770200cb613ca005, scientist
physicien=56b1db8133d9eb398aabd376f07bf8ab5fc584ea0b8bd6a1770200cb613ca005,  scientist
productsales=56b1db8133d9eb398aabd376f07bf8ab5fc584ea0b8bd6a1770200cb613ca005,        sales
servicessales=56b1db8133d9eb398aabd376f07bf8ab5fc584ea0b8bd6a1770200cb613ca005,  sales

Note: it is not possible to specify salt in ini configuration.

Create Logout Page

Any application that have login feature should have also logout feature. Logging out current user with Shiro is easy, use command:

//acquire currently logged user and log him out
SecurityUtils.getSubject().logout();

Logout page then looks like this:

<%@ page import="org.apache.shiro.SecurityUtils" %>
<% SecurityUtils.getSubject().logout();%>
You have succesfully logged out. 


Adding Authorization

We conclude first part by adding authorization to the application. We begin by limiting users access to pages. No user should be able to see pages of other departments. This provides only partial security to the project, since the user is still able to use ‘all application functions’ page or edit URL in the browser to do any action. We will call it page level authorization.

Then, we limit users ability to perform actions themselves. Even if user opens ‘all application functions’ page or edits url in the browser, he will be allowed to perform only functions specific to his department. We will call it function level authorization.

New requirements: user is not able to see pages of departments he does not belong to. User is able to perform only his departmental functions. Only exception to previous rules is administrator, who can perform both administrative and repair functions.

Page Authorization

Page level authorization is done with roles filter. Parameter part of the filter may contain any number of roles. Logged user can access page only if he has all supplied roles.

As usual, roles filter is configured in Shiro.ini file:

[urls]
# force ssl for login page
/simpleshirosecuredapplication/account/login.jsp=ssl[8443],authc

# only users with some roles are allowed to use role-specific pages 
/simpleshirosecuredapplication/repairmen/**=authc, roles[repairman]
/simpleshirosecuredapplication/sales/**=authc, roles[sales]
/simpleshirosecuredapplication/scientists/**=authc, roles[scientist]
/simpleshirosecuredapplication/adminarea/**=authc, roles[Administrator]

# enable authc filter for the all application pages; as Shiro reads urls from up to down, must be last
/simpleshirosecuredapplication/**=authc

Test whether security works: log in as any sales user, click home, click ‘repairmen page’ link. You will see an ugly error.

We finish page authorization and replace error with redirect to an error page. Default Shiro filters have property unauthorizedUrl. In case of unauthorized access, the filter will redirect user to specified url.

[main]
# redirect to an error page if user does not have access rights
roles.unauthorizedUrl = /simpleshirosecuredapplication/account/accessdenied.jsp

accessdenied.jsp:

<body>
Sorry, you do not have access rights to that area.
</body>


Functions Authorization

All departmental pages are secured now. However, any user can still perform any function on ‘all application functions’ page. Moreover, any logged user can edit url and thus do any action. For example, if you log in as sales and put https://localhost:8443/simpleshirosecuredapplication/masterservlet?action=MANAGE_REPAIRMEN into url, the application will perform manage repairmen function too (then it will throw null pointer exception, but security breach was already done).

We assign unique
permission to each function. They are in split into groups:

  • all permissions are in "functions" group,
  • all administrative permissions are in "manage" group,
  • all repair permissions are in "repair" group,
  • all sale permissions are in "sale" group,
  • all science permissions are in "science" group.

Shiro supports multi-level permissions represented as strings. Levels are separated with symbol ‘:’. E.g. "functions:manage:repairmen" has three levels: "functions", "manage" and "repairman". Multi-level permissions allow for easy permissions grouping. For example, science group belongs to functions group and contains three permissions:

  • functions:science:research,
  • functions:science:writearticle,
  • functions:science:preparetalk.

Actions verify logged user permissions before doing their job:

public String doIt() {
    String neededPermission = getNeededPermission();
    // acquire logged user and check permission
    if (SecurityUtils.getSubject().isPermitted(neededPermission))
        return "Function " + getName() + " run succesfully.";

    throw new UnauthorizedException("Logged user does not have " + neededPermission + " permission");
}

NOTE: Another way how to achieve the same goal is through annotations.

The
PerformFunctionAndGoBackServlet servlet catches authorization exception and converts it into error message:

private String performAction(String actionName) {
    try {
        Actions action = findAction(actionName);
        String result = action == null ? null : action.doIt();
        log.debug("Performed function with result: " + result);
        return result;
    } catch (ShiroException ex) {
        log.debug("Function failed with " + ex.getMessage() + " message.");
        return "Error: " + ex.getMessage();
    }
}

Finally, we need to configure permissions to roles in Shiro.ini file. Shiro supports wildcards for multi-level permissions. Thus, we do not have to specify each departmental permission separately:

[roles]
# members of departments should be able to perform all departmental functions
sales=functions:sale:*
scientist=functions:science:*
repairman=functions:repair:*

# administrators are able to do all management functions and repair functions
Administrator=functions:manage:*,functions:repair:*

You can now try functions on ‘all application functions’ page. If logged user does not have required permission, an error message appears on top of the page. Moreover, if you log in as sales and try hacking https://localhost:8443/simpleshirosecuredapplication/masterservlet?action=MANAGE_REPAIRMEN, you will see an error message in the console (instead of success message).

End

Final application is available in
‘static_authentication_and_authorization’ branch on Github.

In the
second part we will create custom realm and move users, passwords, roles and permissions from ini file to database. The
third part is dedicated to Apache Shiro cryptography package.


Reference:
Apache Shiro Part 1 – Basics from our
JCG partner Maria Jurcovicova at the
This is Stuff blog.

Source : http://www.javacodegeeks.com/2012/05/apache-shiro-part-1-basics.html

Advertisements

Java High CPU troubleshooting guide – part 1

This article is part 1 of a series that will provide you with a comprehensive guide on how you can troubleshoot and identify root cause of Java high CPU problems.

This guide is also applicable for standalone Java programs but designed to help individuals involved in day to day Java EE enterprise production support. It will also include the list of the most common high CPU problems along with high level resolution strategies.


Production problems resolution mindset review

Before we go any further, it is important to review your production problem resolution mindset. One of the most common “reflexes” that I have seen in my experience with Java EE production support teams is that Java VM / middleware restart is often the first recovery action that is performed. While a premature restart can quickly eliminate the business impact, it can also prevent you to get all the technical facts; reducing your capability to identify the root cause and exposing the platform to future re-occurrences of the problem.

Before pulling the trigger and shutdown your Java VM process, ask yourself the following question: do I have all the data available to perform a root cause analysis post restart? If the answer is
no then my recommendation to you is to review and improve your current platform monitoring and / or troubleshooting approaches. Proper performance data gathering before and during a high CPU problem is critical.

Java high CPU – what is it exactly?

Now back to our original topic, a high CPU problem is defined by an observation of one or many Java VM processes consuming excessive CPU utilization from your physical host(s). Excessive CPU can also be described by an abnormal high CPU utilization vs. a known & established
baseline. Ex: if the average CPU utilization of your Java VM under peak load condition is 40% then excessive CPU threshold can be set around 80%.

A typical Java VM process contains several Java Threads, some waiting to do work and others currently executing tasks. The # of Threads can be very low in the event of a single Java program and very high for Java EE enterprise platforms processing heavy concurrent transactions.

In order to understand and identify the source of high CPU of one or many of your Java processes, you will need to understand and perform a full breakdown of all Threads of your Java VM so you can pinpoint the biggest contributors. This analysis exercise can be visualized as per below diagram.

Understand your average CPU utilization

As I mentioned in the earlier section, it is very important that you understand your current average CPU utilization, what I refer as a baseline. This is crucial data that needs to be monitored on a regular basis as part of a comprehensive and ongoing platform capacity planning strategy.

Proper understanding and tracking of your average and “healthy” CPU utilization observed from your Java VM processes will allow you to quickly detect abnormal CPU surge scenarios and correlate with possible root causes (problem introduced by a project, unexpected load increase etc.). Finally, this will provide you with proper threshold values necessary to configure pro-active CPU related alerts using monitoring tools of your choice.

Understand your production environment and available tools

As the middleware and / or application support prime, you really need to understand your production environment, including the out-of-the-box tools available for you to perform low level troubleshooting tasks. This may be trivial for some individuals but if you just recently started to work on a new Java or Java EE platform for a new client, my recommendation is that you should spend enough time understand your client’s environment specifications & business situation as per below:

  • Physical & virtual host configuration and capacity (total # of assigned CPU cores, RAM etc.)
  • OS vendor, version and patch level
  • Middleware vendor, versions and patch level
  • Java vendor & versions (including 32-bit vs. 64-bit); including patch level
  • Third party API’s used within the Java or Java EE applications
  • Existing monitoring tools that you can leverage for historical data and trend analysis
  • History of the environment, known issues, resources utilization etc.
  • Business traffic breakdown per application along with average & peak traffic level of the platform; including peak business periods

Collecting all proper facts as per above will definitely help your root cause analysis process going forward; including for high CPU related problems.

Your homework before jumping to the part 2

Before we jump into the part 2 of this CPU troubleshooting guide, I highly recommend that you study and understand each the following articles below. Focus on the ones applicable for your environment. Each of these articles will provide you with a technical step by step on how to breakdown CPU per Thread for a Java VM; a key troubleshooting skill to acquire in order to investigate Java CPU related problems. This technique is common on some aspects and specific depending of the OS.

# CPU per Thread analysis on Solaris


http://javaeesupportpatterns.blogspot.com/2011/12/prstat-solaris-pinpoint-high-cpu-java.html

# CPU per Thread analysis on Linux


http://javaeesupportpatterns.blogspot.com/2012/02/prstat-linux-how-to-pinpoint-high-cpu.html

# CPU per Thread analysis on AIX


http://javaeesupportpatterns.blogspot.com/2011/12/prstat-aix-how-to-pinpoint-high-cpu.html

# CPU per Thread analysis on Windows


http://javaeesupportpatterns.blogspot.com/2012/04/java-thread-cpu-analysis-on-windows.html

I hope this article has provided with a good starting point on Java CPU problems. The part 2 will provide you with the troubleshooting guide which will include flow diagrams allowing you to choose the right investigation path depending of your problem case.


Reference:
Java High CPU troubleshooting guide – part 1 from our
JCG partner Pierre-Hugues Charbonneau at the
Java EE Support Patterns & Java Tutorial blog.

Source : http://www.javacodegeeks.com/2012/05/java-high-cpu-troubleshooting-guide.html

The depths of Java: API leak exposed through covariance

Java can be very tricky some times, especially in API design. Let’s have a look at a very interesting showcase. jOOQ strongly separates API from implementation. All API is in the
org.jooq package, and public. Most implementation is in the
org.jooq.impl package and package-private. Only factories and some dedicated base implementations are public. This allows for very powerful package-level encapsulation, exposing mostly only interfaces to jOOQ users.


A simplified example of package-level encapsulation

Here’s roughly how jOOQ models SQL tables. The (overly simplified) API:

package org.jooq;

/**
 * A table in a database
 */
public interface Table {

  /**
   * Join two tables
   */
  Table join(Table table);
}

And two (overly simplified) implementation classes:

package org.jooq.impl;

import org.jooq.Table;

/**
 * Base implementation
 */
abstract class AbstractTable implements Table {

  @Override
  public Table join(Table table) {
    return null;
  }
}

/**
 * Custom implementation, publicly exposed to client code
 */
public class CustomTable extends AbstractTable {
}


How the internal API is exposed

Let’s assume that the internal API does some tricks with covariance:

abstract class AbstractTable implements Table, InteralStuff {

  // Note, this method returns AbstractTable, as it might
  // prove to be convenient to expose some internal API
  // facts within the internal API itself
  @Override
  public AbstractTable join(Table table) {
    return null;
  }

  /**
   * Some internal API method, also package private
   */
  void doThings() {}
  void doMoreThings() {

    // Use the internal API
    join(this).doThings();
  }
}

This looks all safe at the first sight, but is it? AbstractTable is package-private, but CustomTable extends it and inherits all of its API, including the covariant method override of “AbstractTable join(Table)”. What does that result in? Check out the following piece of client code

package org.jooq.test;

import org.jooq.Table;
import org.jooq.impl.CustomTable;

public class Test {
  public static void main(String[] args) {
    Table joined = new CustomTable();

    // This works, no knowledge of AbstractTable exposed to the compiler
    Table table1 = new CustomTable();
    Table join1 = table1.join(joined);

    // This works, even if join exposes AbstractTable
    CustomTable table2 = new CustomTable();
    Table join2 = table2.join(joined);

    // This doesn't work. The type AbstractTable is not visible
    Table join3 = table2.join(joined).join(joined);
    //            ^^^^^^^^^^^^^^^^^^^ This cannot be dereferenced

    // ... so hide these implementation details again
    // The API flaw can be circumvented with casting
    Table join4 = ((Table) table2.join(joined)).join(joined);
  }
}


Conclusion

Tampering with visibilities in class hierarchies can be dangerous. Beware of the fact that API methods declared in interfaces are always public, regardless of any covariant implementations that involve non-public artefacts. This can be quite annoying for API users when not properly dealt with by API designers.

Fixed in the next version of jOOQ


Reference:
The depths of Java: API leak exposed through covariance from our
JCG partner Lukas Eder at the
JAVA, SQL, AND JOOQ blog.

Source : http://www.javacodegeeks.com/2012/05/depths-of-java-api-leak-exposed-through.html

5′ on IT-Architecture: three laws of good software architecture

The issue with architectural decisions is that they effect the whole system and/or you often need to make them early in the development process. It means a lot effort if you change that decision a couple of months later. From an economic standpoint architectural decisions are often irrevocable. Good architecture is one that allows an architect to make late decisions without superior effect on efforts and costs. Let’s put that on record.

Law 1: Good architecture is one that enables architects to have a minimum of irrevocable decisions.

To minimize the set of irrevocable decisions the system needs to be responsive to change. There is a major lesson I have learned about software development projects: Nothing is permanent except change. The client changes his opinion about requirements. The stakeholders change their viewpoint of what’s important. People join and leave the project team. The fact that change alone is unchanging leads me to the second rule of good architecture, that is:

Law 2: To make decisions revocable you need to design for flexibility.

This is the most provocative statement and I am having controversial discussions here. The reason is that flexibility introduces the need for abstraction. Abstraction uses a strategy of simplification, wherein formerly concrete details are left ambiguous, vague, or undefined (from
Wikipedia). This simplification process isn’t always simple to do and to follow for others in particular. "Making something easy to change makes the overall system a little more complex, and making everything easy to change makes the entire system very complex. Complexity is what makes software hard to change."
from M. Fowler) This is one core problem of building good software architecture: Developing software that is easy to change but at the same time understandable. There are several concepts that try to tackle this paradox problem:
design patterns and
object oriented design principles. Polymorphism, loose coupling and high cohesion are flexibility enablers to me.

Law 3: To make use of flexibility one needs to refactor mercilessly.

Flexibility is not an end in itself. You need to actively make use of flexible design. If something is changing and it makes a previous design or architectural decision obsolete you need to go into the code and change the software. Otherwise the effort of building flexible software is useless and technical debt may cause late delays and a maintenance nightmare. The fact that you take rigorous action on your code base requires continuous feedback about the qualities of your software. To be able to refactor it is therefore essential that the code base is covered by a sufficient amount of automated tests. In an ideal scenario everything is integrated into a continuous integration environment to receive permanent feedback about the health of your code base.



Source : http://www.javacodegeeks.com/2012/05/5-on-it-architecture-three-laws-of-good.html

Programs and Technical Debt

Once you have a program (a collection of interrelated projects focused on one business goal) and you have technical debt, you have a much bigger problem. Not just because the technical debt is likely bigger. Not just because you have more people. But because you also geographically distributed teams, and those teams are almost always separated by function and time zone.

So, my nice example of a collocated team in
Thoughts on Infrastructure, Technical Debt, and Automated Test Framework, rarely occurs in a program, unless you have cross-functional teams collocated in a program. If they do, great. You know what to do.

But let’s assume you don’t have them. Let’s assume you have what I see in my consulting practice: an architecture group in one location, or an architect in one location and architects around the world; developers and “their” testers in multiple time zones; product owners separated from their developers and testers. The program is called agile because the program is working in iterations. And, because it’s a program, the software pre-existed the existence of the agile transition in the organization, so you have legacy technical debt up the wazoo (the technical term). What do you do?

Let’s walk through an example, and see how it might work. Here’s a story which is a composite from several clients; no clients were harmed in the telling of this story.

Let’s also assume you are working on release 5.0 of a custom email client. Release 4 was the previous release. Release 4 had trouble. It was late by 6 months and quite buggy. Someone sold agile as the way to make software bug-free and on-time.

You do not have automated tests for much of the code, unit tests or system tests. You have a list of defects that make Jack the Ripper’s list of killings look like child’s play. But agile is your silver bullet.

The program manager is based in London. The testers for the entire program are in Bangalore because management had previously fired all the testers and outsourced the testers. That was back in release 2. They have since hired all the Bangalore testers as employees of the Bangalore subsidiary. The program architect is based in San Francisco, and there is an architect team that is dispersed into 4 other teams: Denver, LA, Munich, and Paris. The developers are clustered in “Development Centers of Excellence:” Denver, LA, Cambridge, Paris, London, Munich, and Milan. That’s 8 development teams.

Oh, and if you think I’m kidding with this scenario, I’m not. This is what most of my clients with geographically distributed teams and programs face on a daily basis. They deserve your sympathy and empathy. Do not tell them, “Don’t go agile.” That’s nuts. They have a right to go agile. You can tell them, “Don’t go Scrum.” That’s reasonable. Scrum is for a cross-functional co-located team. Agile is for everyone. Scrum is for a specific subset.

What do you do?
  1. Assign specific testers to specific development teams. No calling people resources; that allows managers to treat people like resources and plug-and-play them. You need to get rock-solid teams together. Once you have teams together, you can name them.
  2. Name teams so the teams reflect the feature groups they work on. What does an email product do? It gets email, it sorts email, it deletes, it forwards, it creates new mailboxes, and so on. The eight feature teams had to be named for the feature areas: Platform for the general features, Sort, Delete, Forward. There were two teams who worked on Platform. They were called Platform 1 and Platform 2. At one point, someone suggested they call themselves Thing1 and Thing2 from the Dr. Seuss book.
  3. Make sure you have enough product owners so they can develop roadmaps for each feature area. With a roadmap, the teams know where they are going. Even more importantly, the architects know where the program is going.
  4. Architects think and provide just enough guidance ahead. In a small project, the architecture can probably evolve with the project. In a larger program, that risk is too large. You have too many people developing in parallel for the architecture to evolve on its own with no guidance. But I do not mean there should a Master Architect Who Knows All Handing Down the Architecture From On High. NO NO NO.
    I want the architect who is a working member of the development team, who also is part of an architecture community of practice team, who curates the architecture, who guides the business value of the architecture. I do not want Big Architecture Up Front. But Thinking Up Front? Sure, that’s a great idea. Stuck on only one idea? Bad. Willing to spike an idea? Great. Willing to play in a sandbox and debate several ideas? Great. I wrote about this before, in
    How Agile Architects Lead.
  5. Decide what done means for every feature. You must have acceptance criteria for each feature. What does that mean? You need a product owner present for each team. You still need the conversations with each team to discuss what done means. Especially with a geographically distributed team, you need the conversation when you create the backlog at the beginning of the iteration.
  6. The US development teams had trouble planning their iterations with their testers, because of the time zone differences with the testers. So, they asked their product owners if the product owner would write more than just a few phrases on the cards, because that would help them get through the iteration planning meeting faster. Someone was going to get up early or stay up late, and either way, someone was going to suffer. It made more sense to have a little bit more preparation than less sleep.
  7. Decide to do continuous integration and stick with it. Especially if you know you have technical debt and you don’t want to create more, you have to do continuous integration now. That prevents more technical debt.
  8. I have recommended to some teams that they have one-week iterations so that they stop the estimation nonsense and make their stories small. The point of estimation is so that you have an idea of what you can do as a team and not commit to more than that. The idea is that if you know what it takes to make your stories small, you will.
    Instead, we have all these crazy rituals around estimation and management tracking velocity of all things. (Yes, I’ve been drafting this post for a long time, and I wrote
    Why Does Management Care About Velocity? last week.) You know, velocity is a little like weight. Only you and your doctor need to know your weight. If you are healthy, you are fine. If you are not, you need to change something.If your team velocity is not healthy, you, as a team, need to change it. But, your management has no business butting its head in. Only you can change it.
  9. When you limit the iteration length, you tend to have the team swarm around a story. This is a tendency, not a given. If I really was the Empress of the Universe, I would decree this, but I’m not, so I won’t. If you want to decrease technical debt, or even eliminate it on your program, explain that your team will only work on one story at a time until that story is done. That story will be polished and gleaming. Fast. You will not have to worry about what kinds of testing will be done. All if it will be done.
  10. Explicitly discuss what you will automate for testing and when. In a program, I assume we will have automated system tests first. I assume we will do exploratory tests later. That’s because if you don’t start building something for test automation when you start the program and refactor as you proceed, you can never catch up. I assume every time we fix a defect, we will have an automated test for it. I also assume we build these assumptions into how we develop 🙂

So far, this is all about preventing more technical debt, not what happens when you trip over technical debt as you enter code or tests you never looked at before.

If you expected to walk into a closet, take out a shirt, and close the closet door, that’s one thing. But now, you stepped into something out of one of those death-by-hoarding shows on TV, you have an obligation to do something. You can document the problem as you encounter it; you can let the product owner know; file a defect report; write a test so you can contain the debt; and maybe you have more options. Whatever you do, make sure you have done something. Do not open the door, see the mess inside and close the door on the mess. It’s tempting. Oh my, it is tempting.

See, on programs because of the size, everything is magnified. With more people and more teams, everything is harder. Things happen faster. If you have co-located cross-functional teams, no problem. But if you don’t have co-located cross-functional teams, you have to work with what you have. And, if you already have a big legacy product, you want to address technical debt in small chunks, refactoring in small bits, integrating as you proceed.

My philosophy is this: the bigger the program, the more you need to become accustomed to working in small chunks, integrating as you go. Fully implement a small story, integrate it on the mainline. Everyone on the program does that. If you need help from an integration team, so be it.

But, if everyone only implements small stories, and everyone takes care of their own technical debt as they discover it, you don’t need an army of integration people. You only need an army of integration people when you have technical debt around integration and release. Fix that, and everyone can become responsible for their own integration.

And, if you can’t release, that’s where the architects should start. If you can’t do continuous integration, that’s where the architects should start. Because that’s what’s preventing you from making progress on the product. Work backwards from release, and then the architects can work on the rest of the product. Until you can release and build reliably, the rest of the product doesn’t matter.


Reference:
Programs and Technical Debt from our
JCG partner Johanna Rothman at the
Managing Product Development blog.

Source : http://www.javacodegeeks.com/2012/05/programs-and-technical-debt.html

NetBeans Usability Tips

Java IDEs have come a long way since the
days of
JBuilder (though
JBuilder seemed like a
welcome advance at the time). Today’s Java IDEs (such as
NetBeans,
Eclipse,
IntelliJ IDEA, and
JDeveloper) are very advanced tools that most Java developers embrace for writing significant Java code. As advanced as these IDEs are, they all still have their own quirks and each seems better and easier to use when one understands some key tips (or "tricks") to using that IDE more efficiently. In this post, I look at some tips I have found useful when using NetBeans.


Disabling Automatic Scanning

A problem that can be especially onerous when using NetBeans on a large code base with many related projects open is the occasionally too-frequent automatic scanning that NetBeans performs. This is supposed to only occur intermittently and its intention is good, but sometimes the intended behavior’s value is worth less than the performance-degrading cost justifies. Fortunately, this option can be disabled when its cost is greater than its benefit. In the
NetBeans for PHP blog post
Enable auto-scanning of sources – Scan for External Changes,
Petr Pisl covers how to do this in
NetBeans 6.9.

This feature is also supported in
NetBeans 7.1 as shown in the following screen snapshot (window shown is accessible by selecting Tools ⇒ Options ⇒ Miscellaneous ⇒ Files).

 

Controlling Level of NetBeans Hints

NetBeans’s
Java hints can aid the Java developer in improving and modernizing his or her Java code. The hints cover topics as diverse as performance, safety, conciseness, coding standards, likely bugs, latest JDK standards, and best practices. I do not cover these useful hints in more detail here because I’ve already covered them in multiple previous posts. I introduced NetBeans hints and how to enable them , configure them as warnings or errors, and introduced seven of the most important hints in my blog post
Seven Indispensable NetBeans Java Hints. In the blog post Seven NetBeans Hints for Modernizing Java Code, I discussed seven more hints that are useful for bridging legacy Java code forward to use the best features of newer SDKs (
J2SE 5,
Java SE 6, and
Java SE 7). My post
Creating a NetBeans 7.1 Custom Hint demonstrates writing custom hints to further expand NetBeans hinting capability beyond the out-of-the-box hints.

Setting Source/Target JDK Appropriately

In the blog post
Specifying Appropriate NetBeans JDK Source Release, I looked at several advantages of setting the JDK level for the NetBeans projects’ source/target JDKs appropriately. This can make a major difference for developers using JDK 7 as it helps the hints covered in the previous tip to show areas where pre-JDK 7 code can be migrated to JDK 7 constructs. However, even developers using JDK 6 or JDK 5 can find value to having this set appropriately. The appropriate setting not only advertises features that are available, but it also prevents developers from mistakenly using newer versions when they are not yet available in the actual version of code the developer should be using. NetBeans will warn the developer that certain features are not available for that JDK setting, so it is important to have it set properly.

Leveraging 
NetBeans Keyboard Commands

Whether it’s
vi,
emacs,
Eclipse,
NetBeans, or any other editor, the masters of the respective editors know and frequently use keyboard commands to get work done quickly. NetBeans offers so many keyboard-based commands that it’s difficult to summarize them. However, some good starting points include
Highlights of NetBeans IDE 7.0 Keyboard Shortcuts and Code Templates,
NetBeans Tips and Tricks,
Keyboard Shortcuts I Use All the Time,
NetBeans IDE Keyboard Shortcuts, and
NetBeans Shortcut Keys.
NetBeans even supports
Eclipse key bindings!

Hiding Clutter and Noise with Code Folding

My preference is to have as clean of code as possible. Sometimes, however, I am forced to deal with code that has a lot of unimportant junk or noise in it. In such cases,
NetBeans’s code folding support is welcome because I can hide that noise. It would obviously better if I could remove the unnecessary noise and
code folding can be abused, but I am appreciative of the feature when it’s my only option for reducing clutter and noise so that I can focus on what matters. I discussed
NetBeans code folding in further detail in the post
NetBeans Code Folding and the Case for Code Folding.

Other NetBeans Tips

There are numerous other useful NetBeans tips available online.

Roman Strobl’s NetBeans Quick Tips

In the
blog he maintained while working at
Sun Microsystems,
Roman Strobl wrote several "NetBeans Quick Tip" posts (although dated [mid-2000s], several of these are still applicable):
  1. NetBeans Quick Tip #1 – Setting Target JDK.
  2. NetBeans Quick Tip #2 – Generating Getters and Setters
  3. NetBeans Quick Tip #3 – Increasing Font Size
  4. NetBeans Quick Tip #4 – Extending the Build Process
  5. NetBeans Quick Tip #5 – EOL Sweeper
  6. NetBeans Quick Tip #6 – Abbreviations in Editor
  7. Quick Tip #7 – Macros in Editor
  8. NetBeans Quick Tip #8 – Using Custom Folds
  9. Quick Tip #9 – Better Responsivenes of Error Marks and Hints
  10. NetBeans Quick Tip #10 – Diffing Two Files
  11. NetBeans Quick Tip #11 – How to Save As…
  12. Netbeans Quick Tip #12 – Fast Navigation to Methods and Fields
  13. NetBeans Quick Tip #13 – Define a Shortcut for Ant Target
  14. NetBeans Quick Tip #14 – Accessing Files Outside Projects
  15. NetBeans Quick Tip #15 – Adding Multiple Components with Matisse
  16. NetBeans Quick Tip #16 – Using Dependent Projects
  17. NetBeans Quick Tip #17 – Faster Building of Projects with Dependencies
  18. NetBeans Quick Tip #18: What to Do when Things Go Wrong?
  19. NetBeans Quick Tip #19 – Positioning without Guidelines in Matisse
  20. NetBeans Quick Tip #20 – Killing Processes
  21. NetBeans Quick Tip #21 – Achieving Same Size
  22. NetBeans Quick Tip #22 – Using Matisse’s Connection Manager
  23. NetBeans Quick Tip #23 – Changing Code in Blue Guarded Blocks
  24. NetBeans Quick Tip #24 – Correct Javadoc
  25. NetBeans Quick Tip #25 – Case Insensitive Code Completion
  26. NetBeans Quick Tip #26 – Short Package Names
  27. NetBeans Quick Tip #27 – Implementing Abstract Methods
  28. NetBeans Quick Tip #28 – Configuring Derby Database in NetBeans 5.0
  29. NetBeans Quick Tip #29 – Monitoring HTTP Communication
  30. NetBeans Quick Tip #30 – When GroupLayout Fails
  31. NetBeans Quick Tip #31 – Changing the Look and Feel
  32. NetBeans Quick Tip #32 – Faster and More Stable Ruby Support
  33. NetBeans Quick Tip #33 – Show Error Using Keyboard
  34. Keyboard Shortcuts I Use All the Time
Other Posts on NetBeans Tips

Your Favorite NetBeans Tip or Trick?

What is your favorite NetBeans tip or trick?


Reference:
NetBeans Usability Tips from our
JCG partner Dustin Marx at the
Inspired by Actual Events blog.

Source : http://www.javacodegeeks.com/2012/05/netbeans-usability-tips.html

How to use RestTemplate with Basic Authentication in Spring

This article shows how to use Springs
RestTemplate to
consume a RESTful Service secured with Basic Authentication; the article is part of an in depth
series on REST with Spring.

The REST with Spring series:


Setting up the RestTemplate in Spring

Bootstraping the
RestTemplate into the Spring context can be done by simply declaring a bean for it; however, setting up the
RestTemplate with
Basic Authentication will require manual intervention, so instead of declaring the bean directly, a Spring
FactoryBean will be used for more flexibility. This factory will create and configure the template on initialization:

@Component
public class RestTemplateFactory implements
    FactoryBean< RestTemplate >, InitializingBean{
   private RestTemplate restTemplate;

   public RestTemplate getObject(){
      return restTemplate;
   }
   public Class< RestTemplate > getObjectType(){
      return RestTemplate.class;
   }
   public boolean isSingleton(){
      return true;
   }
   public void afterPropertiesSet(){
      restTemplate = new RestTemplate(
       new HttpComponentsClientHttpRequestFactoryBasicAuth
        ( new HttpHost(host, port, protocol) );
   }
}

The
host and
port values should be dependent on the environment – allowing the client the flexibility to define one set of values for integration testing and another for production use. The values can be managed by the
first class Spring support for properties files.

Basic Authentication for the RestTemplate

Once Basic Authentication is set up for the template, each request will be sent
preemptively containing the full credentials necessary to perform the authentication process. The credentials will be encoded and will use the
Authorization HTTP Header, in accordance with the specs of the Basic Authentication scheme. An example would look like this:

Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==


Manual management of the Authorization HTTP header

The process of creating the
Authorization header is relatively straightforward for Basic Authentication, so it can pretty much be done manually with a few lines of code:

HttpHeaders createHeaders( String username, String password ){
   return new HttpHeaders(){
      {
         String auth = username + ":" + password;
         byte[] encodedAuth = Base64.encodeBase64( auth.getBytes() );
         String authHeader = "Basic " + new String( encodedAuth );
         set( "Authorization", authHeader );
      }
   };
}

Then, sending a request becomes just as simple:

restTemplate.exchange
 (uri, HttpMethod.POST, new HttpEntity<T>(createHeaders(username, password)), clazz);


Automatic management of the Authorization HTTP header

Both Spring 3.0 and 3.1 have very good support for the Apache HTTP libraries:

  • Spring 3.0, the CommonsClientHttpRequestFactory integrated with the now end of lifed HttpClient 3.x
  • Spring 3.1 introduced support for the current HttpClient 4.x via HttpComponentsClientHttpRequestFactory (support added in the JIRA SPR-6180)

Let’s start setting things up with HttpClient 4 and the newly introduced support from Spring 3.1. The
RestTemplate accepts the HTTP request factory in it’s constructor – allowing a factory that supports Basic Authentication to be passed into the template – so far, so good. However, using the existing
HttpComponentsClientHttpRequestFactory will prove to be difficult, as the architecture of
RestTemplate was designed
without good support for
HttpContext – an instrumental piece of the puzzle.

Fortunately this is not without a solution, but first, some details on how to do Basic Authentication with the HttpClient library itself. Doing preemptive Basic Authentication with HttpClient 4.x is
a bit of a burden: the authentication info is cached and the process of setting up this authentication cache is very manual and unintuitive:

private HttpContext createHttpContext() {
    // Create AuthCache instance
    AuthCache authCache = new BasicAuthCache();
    // Generate BASIC scheme object and add it to the local auth cache
    BasicScheme basicAuth = new BasicScheme();
    authCache.put(targetHost, basicAuth);
    // Add AuthCache to the execution context
    BasicHttpContext localcontext = new BasicHttpContext();
    localcontext.setAttribute(ClientContext.AUTH_CACHE, authCache);
    return localcontext;
}

HttpResponse response = httpclient.execute
 (targetHost, new HttpGet("/"), createHttpContext());

Now, this
HttpContext needs to be hooked up to the
RestTemplate, so that requests created via the template can benefit from the Basic Authentication support. As mentioned before, this is not an easy as it should be – the entire
HttpComponentsClientHttpRequestFactory needs to be
subclassed and the
createHttpContext method
overridden:

public class HttpComponentsClientHttpRequestFactoryBasicAuth
 extends HttpComponentsClientHttpRequestFactory {
   HttpHost host;

   public HttpComponentsClientHttpRequestFactoryBasicAuth(HttpHost host) {
      super();
      this.host= host;
   }
   protected HttpContext createHttpContext(HttpMethod httpMethod, URI uri) {
      return createHttpContext();
   }
   private HttpContext createHttpContext() {
      ...
   }
}

And with that, everything is in place – the
RestTemplate will now be able to support the Basic Authentication scheme; a simple usage patter would be:

HttpComponentsClientHttpRequestFactory requestFactory =
 (HttpComponentsClientHttpRequestFactory) restTemplate.getRequestFactory();
DefaultHttpClient httpClient =
 (DefaultHttpClient) requestFactory.getHttpClient();
httpClient.getCredentialsProvider().setCredentials(
 new AuthScope(host, port, AuthScope.ANY_REALM),
  new UsernamePasswordCredentials("name", "pass"));

For an in depth discussion on how to secure the RESTful Service itself,
check out this article.

Maven dependencies

The following Maven dependencies are required for the
RestTemplate itself and for the HttpClient library:

<dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-webmvc</artifactId>
   <version>3.1.1.RELEASE</version>
</dependency>

<dependency>
   <groupId>org.apache.httpcomponents</groupId>
   <artifactId>httpclient</artifactId>
   <version>4.1.3</version>
</dependency>

Optionally, if the HTTP
Authorization header is constructed manually, then an additional library is required for the encoding support:

<dependency>
   <groupId>commons-codec</groupId>
   <artifactId>commons-codec</artifactId>
   <version>1.6</version>
</dependency>


Conclusion

Although the 3.x branch of development for Apache HttpClient has reached end of life for a while now, and the Spring support for that version has been fully deprecated, much of the information that can be found on
RestTemplate and security still doesn’t account for
the current HttpClient 4.x releases. This article is an attempt to change that through a detailed, step by step discussion on how to set up Basic Authentication with the
RestTemplate and how to use it to consume a secured RESTful Service.

For the next article, the focus will be on setting up
Digest Authentication in the same way. To go beyond the code samples in the article with a production ready implementation of both the consuming side, examined here, but also the actual RESTful service, check out the
REST github project.


Reference:
How to use RestTemplate with Basic Authentication in Spring from our
JCG partner Eugen Paraschiv at the
baeldung blog.

Source : http://www.javacodegeeks.com/2012/05/how-to-use-resttemplate-with-basic.html