Showing posts with label Technical Tips. Show all posts
Showing posts with label Technical Tips. Show all posts

Wednesday, September 30, 2009

WCI: Varpacks and Dynamic Reload

Varpacks stands for Variable Packages…As their names show, varpacks are portal objects that do 2 main things:

  • Load values from a file into the portal memory (xml format is preferred since there is already a XMLBaseVarPack class that helps load simple XML files)
  • Help access these loaded in memory values from anywhere within the portal application – very likely from your portal “customization” code (i.e. custom activity space, custom view, PEIs, )

The goal here is not to re-explain what Varpacks are, since it is already pretty well explained in the ALUI/WCI developer documentation (4 pages):

http://download.oracle.com/docs/cd/E13174_01/alui/devdoc/docs60/Customizing_the_Portal_UI/Using_Varpacks/plumtreedevdoc_customizing_varpack_intro.htm

The goal here is to explain the following 2 points:

  1. Portal does not have always have to be restarted to load the new Varpack values from the XML file.
  2. Developed properly, a varpack can load/access “experience definition” specific values, which can be very useful when you portal powers multiple portal sites, each with their own different settings etc…

1 - First, to allow you varpack to be reloaded dynamically without the need to restart the portal, you simply need to specify it in your varpack code… and really, it is as simple as overriding the following method:

public override bool CanReloadVarPackFromUI(){
return true;
}

If the method returns true, then the varpack can be reloaded from the UI, otherwise, it cannot…and then can be reloaded only at portal startup (hence requiring portal restart if you change a value in your varpack XML file)…


And what is this UI? It is the fairly unknown “MemoryDebug” activity space (see this article from Function1 that explains how to access it: http://www.function1.com/site/2007/06/alui-portal-memory-debug-page.html).


Now, if you did everything correctly with your varpack development and deployment, you should see in the “MemoryDebug” screen your varpack name in the list of loaded varpack, and a “view” button by it…And if you returned “true” on the overridden CanReloadVarPackFromUI method as shown above, then a “Reload” button will also show up. As you imagine, clicking “reload” will launch the process that loads the values from your varpack XML file into portal memory, hence reloading your new Varpack values without having to restart the portal.


2 – Secondly, let’s imagine your single portal environment powers multiple portals, each with their own style, colors, hostname…etc…well it is easy to imagine, really: that’s what we do almost every time, and that’s what portal is for.


But now, let’s imagine that you built these customization that should be loaded only within portal site A, but not within portal site B…In other word, you want to selectively load your customizations based on criteria such as portal hostname or experience definition ID…mmmmm that is not really possible since your customizations (view replacement, PEIs, custom activity space, etc…) are dynamically loaded when the portal starts and activated directly once loaded…


But with the use of a properly developed Varpacks, you could actually achieve the above scenario easily…


What you need to do is to make your Varpack aware of the experience definition you are currently in, and then define in your XML file variable names suffixed with an experience definition ID.


Here is a sample XML file:


<?xml version="1.0" encoding="UTF-8"?>
<MyCustomVarpack>
<Section1-200>
<mycustomkey1 value="" />
<mycustomkey2 value="" />
<mycustomkey3 value="" />
<mycustomkey4 value="" />
</Section1-200>
<Section2-201>
<mycustomkey1 value="" />
<mycustomkey2 value="" />
<mycustomkey3 value="" />
<mycustomkey4 value="" />
</Section2-201>
</MyCustomVarpack>

As you can see, the sections have an ID appended to it…this ID represents the experience definition ID where the values within this section will be applicable….And as you can see, using this name formatting, you can potentially define completely different values for experience definition 201.


Now, how to make your Varpack class aware of the current experience ID you are in…well you simply need to store the experience definition ID in an instance variable of your varpack object…see the code below:


public class MyCustomUIVarPack : XMLBaseVarPack
{
private static OpenLogger log = OpenLogService.GetLogger("MyCustomUIVarPackLibrary",typeof(MyCustomUIVarPack ));

private int m_ExpId = -1;
private void SetExperienceId(int expDefId)
{
m_ExpId = expDefId;
}
... ...
... ...
}

Then, to assign the current experience definition ID to this instance variable, you need to find the experience definition ID you are in using the call: TaskAPIServerSubPortal.GetSubPortalCachedObject(oUserSession)…


And then, simply assign the ID to the variable using the defined setter SetExperienceId(int expDefId)…here is the code:


//getting the varpack instance
MyCustomUIVarPack customVarpack = (MyCustomUIVarPack)vPackMgr.GetVariablePackage(MyCustomUIVarPack.VARPACK_ID);

//getting the experience definition using the user session
IPTSubPortalInfo objSubportalinfo = TaskAPIServerSubPortal.GetSubPortalCachedObject(oUserSession);
If(null != objSubportalinfo)
{
Log.Debug("Subportal Info Object ID is: {0}",objSubportalinfo.GetObjectID());
customVarpack.SetExperienceId(objSubportalinfo.GetObjectID());
}

Ok, with that done, your varpack object is now “experience definition aware”…and as such, you can simply override the Varpack GetValueAsString / GetValueAsBoolean / GetValueAsInt methods, appending the experience definition ID to the varpack key you want to get the value from.


I hope this article will give you a couple of new ideas in regards of Varpacks…but really will make you realize (if not already) that the WCI Varpack framework is definitely pretty cool and powerful.


As always, let me know what you think…and let me know if you used varpacks to implement other cool use cases…

Monday, May 11, 2009

ALUI Webcenter Grid Search: Maintaining the search cluster repository without loss of service

Sometimes, for maintenance reasons, the cluster repository has to be unavailable for a short amount of time (I.e. the NAS on which it is hosted is patched and need to be restarted, or the central search cluster repository needs to be moved from one server to another, etc…).

The main problem is that search relies heavily on the availability of search cluster repository in order for the portal content to be properly indexed (for example, the cluster registers any new index delta and ensures that these delta are redistributed to each node’s local index, that way the search nodes are always in sync with one another). Unfortunately, I’ve noticed many times that when the cluster becomes not available for as short as a couple of seconds, the search infrastructure does NOT handle this gracefully…

At best, the nodes all go in “read-only” mode automatically (meaning the nodes act only as query service instead of query+index service), corrupting most of the time the process handles of the node being the “indexer” at the time of disconnection (you’ll see “invalid handles” errors in the search status screen for example)…at worse, all the nodes shutdown with a great “out of memory” error. Both scenario is not a good one, since it will not go back in run mode automatically after the disruption is over, and will probably require a overall restart of nodes.

If you need to do this in PROD where uptime is usually a strong requirement, then the idea is to do such an operation without jeopardizing the search capabilities of your portal site(s). Indeed, search being so central to portal, when down, many portlets or components relying on search will be down, and overall service will be pretty degraded.

Fortunately, search comes with a powerful admin utility: cadmin.exe. You can find it on any of your search nodes, usually at the following path:

<pt_search_home>/bin/native/cadmin.exe

Using the tool, you can gracefully put all the search nodes in “read-only” mode before the maintenance operation. Indeed, when the nodes are in read-only mode, each node act as a “disconnected” query service, providing search results solely based on their local index. While in that state, the search cluster can be fully unavailable… and apart from any new content not being indexed, end users will not see any search disruption.

So here are the commands you would want to perform, either manually or in a batch:

cadmin runlevel readonly –-> this puts all the nodes of the cluster in readonly mode
cadmin status –-verbose –-> this give you the status of the cluster. useful to make sure the previous operation worked as expected.

…perform your maintenance operation…

cadmin runlevel run –> put all the nodes on the cluster back in run mode
cadmin status –verbose –> sanity check…

What you would want to do before any search maintenance is perform a search checkpoint (in other word search backup). 2 options: doing it manually using the Admin UI, or using the cadmin tool as follows:

cadmin checkpoint --create

As an extension of this, you can check out the other operations you can perform with the cadmin tool (pretty much everything you can do with the search cluster admin UI, with more power added to it) by entering cadmin –help

Hope that helps. Until next time, take care!

Sunday, March 15, 2009

ALUI Publisher: Easy bug fix with portlet templates…

I know ALUI Publisher is on the down slope, but until then, it does not mean I should not share some of my findings…Lately, I found something in publisher that does not make sense to me. It has to do with the creation of portlets based on publisher portlet templates. Let me explain the problem…and then the quick fix…

The problem:

Out of the box, when you create a portlet based on a publisher portlet template (i.e. announcement, news, etc…), there is a screen where you are asked to choose a publisher folder where the portlet publisher content should be…When clicking on the “chose publisher folder”, a publisher tree pops up, and you can pick the publisher folder. The tree picker will only show you the folders where you have “producer” access…until now it makes sense since only the “producer” role and above can create folders in publisher.

But what if a user does have “producer” access to a subfolder Z located in the tree structure at X > Y > Z…but does not have producer access to the parent folder X and Y?? Then the tree picker would simply stop showing the tree at X, hence not showing the subfolder to which the user has actually access to…hence problem.

A perfect (and probably common) use case is where you have various publisher sites (i.e. an intranet site, and internet site, etc…) and within each of these, you have various “community-related” publisher folders… (folders that contain the web content of each community). You will want your community administrators (in the portal) to also have “producer” or “folder administrator” role in their “community-related” publisher folder…but not have these roles in the parent publisher folders…they should have “reader” access on these parent folders…

The solution:

By reverse engineering publisher one more time, I found out that this problem can be fixed really easily…basically the publisher tree picker is opened with the following url:

“../folderpicker_frame.jsp?sid="+sessionId+"&showItemCategory=0&itemId="+parentFolderId+"&isMultiSelect=false&rootIsCheckable=true&minRoleId=12”

What is interesting in this url is the last parameter: minRoleId=12. What it probably means is “show only the folders to the users who have at least a role 12 – producer role – assigned to it.” That’s it, we have our solution…By removing this parameter altogether, the tree will now show the folders to which the user has minimum access to (“reader” access), hence fixing the user case explained above:

A producer in folder X > Y > Z will now be able to browse down to the Z folder, where he will be able to create a folder.

And don’t worry, it does not impact core security at all. If the user tries to select folder X or Y as a container for his “announcement” portlet, he will simply receive an error message saying ”you do not have enough access to create a folder”…so no problem.

Detailed instructions:

  1. Make a backup of the publisher application files \bea\alui\ptcs\6.4\webapp\ptcs.ear and \bea\alui\ptcs\6.4\webapp\ptcs.war (obvious, no?)
  2. Unpack the publisher archive (ptcs.war)
    1. navigate to \bea\alui\ptcs\6.4\webapp folder
    2. create a new dir: ptcs
    3. navigate to that new dir and execute the following jar command: jar –xvf ../ptcs.war
  3. edit the extracted file: ./portlet_packages/portlet_create.jsi
  4. Find the JavaScript functions “ChooseParentFolder” and “NewParentFolder”
  5. Remove the “minRoleId=12” from the “var url = …” line (1st line in each functions)
  6. Still in \bea\alui\ptcs\6.4\webapp\ptcs folder, repackage the war by executing: jar -cvf ptcs.war *
  7. move the newly created war to the \bea\alui\ptcs\6.4\webapp folder: move ptcs.war ../
  8. navigate to \bea\alui\ptcs\6.4\webapp: cd ../
  9. update the ptcs.ear file with the following command: jar -uvf ptcs.ear ptcs.war

That’s it…

Before restarting publisher, clean the temp files from the publisher container folders just to make sure your modification is loaded properly.

  • \ptcs\6.4\container\tmp\deploy
  • \ptcs\6.4\container\work\jboss.web\localhost

As usual, do this at your own risk…and test it well before deploying to production :)

Sunday, November 16, 2008

ALUI / WebCenter Interaction: Introduction to Native API development (Part 1)

Most of the time, creating a remote portlet application using the standard IDK is enough. Indeed, commonly, you just need to access user profile information, portlet ID, page ID etc... or perform simple operations exposed by the IDK PRC API.

But if you hit the limits of the IDK, it is not the end of the road. In the ALUI/Webcenter development documentation, they explain how to perform various UI customizations such as view replacement, PEI development, Activity Spaces creation or extensions, etc... All these are created within the portal application itself...Another option is to create what I call a "Native API Portlet Application". As its name indicates, this portlet application accesses directly the portal API and can perform virtually any operation that the portal can do...very powerful (but non supported :) so make sure your application is well written so that you can change the portal API access layer easily if you upgrade the portal version for example)

First, the 2 main requirements in order to create a Native API application are:

  • The application needs to be on the same server as Portal, or Automation, or API service
  • The application needs to reference various portal DLLs (located in /ptportal/6.1/bin/assemblies for dotnet, or /ptportal/6.1/lib/java for java). You don’t need them all. Here is the minimum list:



    ptportal\6.1MP1\bin\assemblies\opencache.dll

    ptportal\6.1MP1\bin\assemblies\openconfig.dll

    ptportal\6.1MP1\bin\assemblies\opencounters.dll

    ptportal\6.1MP1\bin\assemblies\openfoundation.dll

    ptportal\6.1MP1\bin\assemblies\openhttp.dll

    ptportal\6.1MP1\bin\assemblies\openkernel.dll

    ptportal\6.1MP1\bin\assemblies\openkernelsearch.dll

    ptportal\6.1MP1\bin\assemblies\openkernelsearchimpl.dll

    ptportal\6.1MP1\bin\assemblies\openlog-framework.dll

    ptportal\6.1MP1\bin\assemblies\openprocman.dll

    ptportal\6.1MP1\bin\assemblies\opensharedcache.dll

    ptportal\6.1MP1\bin\assemblies\opentempfile.dll

    ptportal\6.1MP1\bin\assemblies\openusage.dll

    ptportal\6.1MP1\bin\assemblies\openusage-api.dll

    ptportal\6.1MP1\bin\assemblies\openusage-impl.dll

    ptportal\6.1MP1\bin\assemblies\plumtreeserver.dll

    ptportal\6.1MP1\bin\assemblies\portal.dll

    ptportal\6.1MP1\bin\assemblies\pthome.dll

    ptportal\6.1MP1\bin\assemblies\ptportalobjects.dll

Important note: It is perfectly allowed to also include the IDK Dlls here too... and I find it particularly recommended if you are writing a native API portlet (for instance getting the current login token for example is fairly easy using the IDK API)

Second, you need to create the native session (everything starts from here, really):

1: String strServerConfigDir = ConfigPathResolver.GetOpenConfigPath(); 
2: IOKContext configContext = OKConfigFactory.createInstance(strServerConfigDir,"portal");
3: PortalObjectsFactory.Init(configContext);
4: IPTSession ptsession = PortalObjectsFactory.CreateSession();

From there you actually need to connect this session with a particular user identity. That way, the session object will be aware of your identity and especially the security and permission associated with your identity. In other words, even if you are using the native API, you cannot do more than you are allowed to.

To connect, 2 main options:

  • Use the login token that you can simply get from the IDK API (that is prefered option if you can)
1: String loginToken = m_portletRequest.GetLoginToken(); //IDK call
2: ptsession.Reconnect(loginToken);
  • Use explicit Username (or user ID) / Password to connect.

1: String username = "myusername";
2: String pwd = "mypassword";
3: ptsession.Connect(username, pwd, null);

Third (and Final), you access the right ObjectManager Class depending on which type of portal object you want to interact with.

When the session is created (that was pretty easy, right?), that is when you can actually start interacting with the portal internals...and a majority of actions goes through the PTObjectManager objects (IPTObjectManager interface). For instance:

1: IPTCommunityManager ptCommManager = ptSession.GetCommunities(); //for community objects
2: IPTPageManager ptPageManager = ptSession.GetPages(); //for community page objects
3: IPTObjectManager ptGadgetManager = ptSession.GetGadgets(); //for portlet objects
4: //etc...

You noticed that for portlets, there is not a IPTGadgetManager insterface....that’s ok as all the "Manager" interfaces are children of the base IPTObjectManager interface.

Although the vast majority of manager objects are accessible through a direct ptsession.Get() call, you can also get the right manager using the generic call below (using the classID of the object type you want)

1: IPTObjectManager ptCrawlerManager = ptSession.GetObjectManagers(PT_CLASSIDS.PT_CRAWLER_ID); 
2: //using     the class id of the object type you want

From there, the freedom is yours... and various operations will be available depending on the manager you called. One call for instance that all managers have is the open object:
1: ptObjectManager.Open(objectid, lockObject);

With this call, you get directly access a particular object in the portal, and interact with it as if you were in the UI

In the next article, I will show you how I package these API call in a standalone library to minimize as much as possible any API call within the presentation layer code...

Monday, October 27, 2008

ALUI Publisher - Part 3: No Redirect Bug Fix of Bug Fix :)

I have been relatively lazy in regards of my blog lately, and I have plenty of articles that are stacking up...But in the meantime, I thought this one is pretty urgent.

In one of my previous article, ALUI Publisher - Part 2: Increase Performance by enabling REAL Caching - No Redirect Bug Fix, I was explaining how to fix the Publisher published_content_noredirect.jsp (see  in that serie the benefits of using this instead of published_content_redirect.jsp).

Well, I found out a small little bug on my part, and it is definitely worth fixing if you have not done so already. In my corrective code, I was trimming the content string from its end spaces (just to optimized the HTML output)...and I did not think of the likely negative effects, such as if the buffered content does finish on a meaningful space character (such as a sentence separator etc...)

So the updated code with bug fix is: (specifically the trim that is removed)

//if there is content, forward to the requesting client
int buffersize = 2000;
int charread = 0;
char[] content;

//read until no byte is found in the input stream because request content length is no reliable
// UTF-8 is necessary
BufferedReader bisr = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));

do {
content = new char[buffersize];
charread = bisr.read(content);
out.write(new String(content).trim());
} while(charread > -1);

bisr.close();
bisr = null;
content = null;


Better to find it late than never! :)

The code should be updated on the famous (yeah right...) ALUI Toolbox Google project (or should be soon)

So Long.

Tuesday, July 15, 2008

ALUI Tool: URL (or text) Migration within Publisher Items

Following my previous article "ALUI Administration Tool for Environment Refresh: String Replacing for URLS" talking about migration between environments, here is an extra piece that you might find very useful (I surely use it all the time)

Basically, as explained in the previous article, it is common to have different DNS aliases set up per environment...I.e for publisher remote server, you could have:

Similarly, the publish browsing URL is not an exception to this rule:

  • http://publisher-content.domain.com/publish  for production
  • http://publisher-content-stg.domain.com/publish for staging
  • http://publisher-content-dev.domain.com/publish for development
  • When you add an image or a link in the free text editor of content items in publisher, it will most of the time create an absolute URL to that resource...thus you can imagine that there will be a lot of DNS aliases within a lot of publisher items throughout the environment.

    What happen when you migrate the publisher DB from one environment to another? Well you will have a lot of DEV dns aliases within your Staging environment (in case of a DEV promotion to Stage); or a lot of production DNS aliases within your dev environment in the case of a production refresh to DEV.

    In my previous article "ALUI Administration Tool for Environment Refresh: String Replacing for URLS", I was mostly talking about migrating URL within portal objects, but nothing really about migrating urls within publisher items.

    Thus, I created some DB scripts (SQL Server only for now) that do just that...

    1. puburls-PTCSDIRECTORY-nvarchar-replace.sql: Script to change a particular string within the PUBLISHEDTRANSFERURL and PUBLISHEDURL columns (which is mapped out in the DB to a column of type VARCHAR)
    2. puburls-PTCSVALUE-ntext-replace.sql and puburls-PTCSVALUEA-ntext-replace.sql: Scripts to change a particular string within the "long text" property of a publisher item (which is mapped out in the DB to a column of type TEXT)
      1. PCSVALUES.LONGVALUE (hosting the long text of the currently published item)
      2. PCSVALUESA.LONGVALUE (hosting the long text values of all the previous versions of the item)

    For the first script, PUBLISHEDTRANSFERURL and PUBLISHEDURL columns are of type VARCHAR and thus it is easy to replace a string within those columns using the REPLACE MS SQL Function. Thus, a simple SQL statement is good here.

    The main challenge was really with the 2nd scripts...indeed, within a column of type TEXT, the SQL "REPLACE" function cannot be used...The workaround is to use the PATINDEX and UPDATETEXT functions within a Transact-SQL (T-SQL) script. To give the credit to to the right person, I adapted a script that I found at ASP FAQ - How do I handle REPLACE() within an NTEXT column in SQL Server?

    DISCLAIMER: ALTHOUGH I PERSONNALY USE THIS SCRIPT ALL THE TIME, THERE IS NO GUARANTY; SO USE THIS TOOL AT YOUR OWN RISK blah blah blah AND USE IT ONLY IF YOU ARE PROFFICIENT ENOUGH WITH ALUI PORTAL TECHNOLOGIES.

    Attached is the zip file package that contains the 3 scripts:

    Don't forget to change the string to look for, and the string to replace it with

    puburls-PTCSDIRECTORY-nvarchar-replace.sql

    UPDATE [dbo].[PCSDIRECTORY]
    SET
    [PUBLISHEDTRANSFERURL]=REPLACE([PUBLISHEDTRANSFERURL],'-DEV.DOMAIN.COM','-TST.DOMAIN.COM'),
    [PUBLISHEDURL]=REPLACE([PUBLISHEDURL],'-DEV.DOMAIN.COM','-TST.DOMAIN.COM')
    WHERE
    publishedtransferurl like '%-DEV.DOMAIN.COM%'
    or publishedurl like '%-DEV.DOMAIN.COM%'



    puburls-PTCSVALUE-ntext-replace.sql and puburls-PTCSVALUEA-ntext-replace.sql



    SET @oldString = N'por-pubcontent-dev.domain.com'; -- remove N 
    SET @newString = N'por-pubcontent-tst.domain.com'; -- remove N


    That's it! Let me know if you find it as useful as I do! Enjoy!!

    Monday, June 30, 2008

    ALUI Publisher - Part 2: Increase Performance by enabling REAL Caching - No Redirect Bug Fix

    Previously (http://fsanglier.blogspot.com/2008/06/alui-publisher-part-2-increase.html) , I've been talking (and proving I hope) that using a "no redirect" mechanism for serving published content from publisher is the best option to enable portal caching. Publisher 6.4 offers already such a possibility (although not publicized a lot): using in the portal Published Content Web Service object published_content_noredirect.jsp instead of the standard published_content_redirect.jsp.

    Unfortunately, if you start using this, you are going to start seeing a weird behavior: the publish content is getting truncated in some special cases...and this is due to the way the JSP has been coded. Several options for you: either you wait for a Critical fix to be issued to you by BEA (i am not aware of one yet), or you upgrade to ALUI 6.5 (I hear that this has been fixed in 6.5...have not verified though), or you simply do it yourself, as this is a simple fix to implement (ultimately, that might be the same type of code that would be issued by a CF I imagine)

    By looking at the JSP within the publisher web application archive (ptcs.war - explode the war using jar command), we can see what's wrong and why the content is truncated in some case:

    HttpURLConnection conn = (HttpURLConnection)url.openConnection();

    // make the request
    conn.connect();

    //read the content length
    int contentLength = conn.getContentLength();

    //if there is content, forward to the requesting client
    if( contentLength > 0 ){
    // UTF-8 is necessary
    InputStreamReader isr = new InputStreamReader(conn.getInputStream(), "UTF-8");
    char[] content = new char[contentLength];
    isr.read(content);
    isr.close();
    out.write(content);
    }


    As you can see, an HTTP GET request is made, and the content length of the response is gotten from the "getContentLength()" method. This call is going to get the content length number fro mthe response header rather than actually count all the bytes that are contained in the response content. Thus, since the code base itself on this number to output the content to the JSP output stream (see above: char array of length equal to contentlength), the content will indeed be truncated if the contentlength number is not correct...



    A simple correction (and more robust code) is actually to make sure ALL the content is pushed to the output stream, independently from the contentlength number returned by the response header. Here is my code below that fixes that issue, and also increase performance by using the preferred BufferedReader wrapper class instead of the bare InputStreamReader:




    ------EDITED 3/12/2009--------
    BufferedReader bisr = null;
    try {
    bisr = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
    String line;
    while ( (line = bisr.readLine( ) ) != null ) {
    out.println(line);
    }
    }
    catch(Exception exc){
    throw exc; //to be caught by the global try catch
    } finally {
    if(bisr != null)
    bisr.close();
    bisr = null;
    }
    return;
    ------END EDITED 3/12/2009--------

    Basically, the code will read by chunks of 2000 chars (this is a chunk size that I think is appropriate) the entirety of the content until the last character...and write it all to the output stream...This does not rely on the contentlength at all, and thus is more reliable and robust.



    After changing the published_content_noredirect.jsp as above, you can repackage the ptcs.war with the new corrected JSP (within the root of the previously extracted ptcs.war folder, run jar -cvf ptcs.war * command) and redeploy to ALL redirector and publisher instances...



    Voila, you have your perfect solution for ALUI 6.1 and Publisher 6.4 (and previous versions too).

    ALUI Publisher - Part 2: Increase Performance by enabling REAL Caching

    In my previous post http://fsanglier.blogspot.com/2008/02/alui-publisher-increase-performance.html(man, already couple of month ago...I know I've been sucked into a black hole since then :) ) I was talking about how to best design a scalable and redundant ALUI Publisher architecture. But what I had not pushed enough in the last article was performance.

    In my opinion, the Publisher standard behavior for serving content has some performance flaw (at least in ALUI 6.1 - Publisher 6.4) related to caching... Since there are ways around this (that's what we do, right?), I think you might benefit from this a lot, and in my last implementation, the change explained below increased performance under load to complete new levels (in the meantime reducing DB requests and DB and publisher redirector CPU utilization)...So here it goes...

    As I explained in previous post, each published content portlet within a portal page are going to make a request to the Publish Content Redirector component, and particularly the JSP page in charge of performing the redirect: published_content_redirect.jsp (you can see that defined in the Published Content portal web service object). As the name says, this Java Server Page (JSP) performs a 302 redirect to the published content item (as seen in previous post, this location should be served by your favorite web server, apache or IIS for example...). But BEFORE making this redirect, it must know where to redirect to... so for that, the code makes a DB request to the publisher Database in order to get the browsing path to the published content item (passing the publisher content item ID that was saved when you created your published content portlet earlier).

    Ok so here is the first flaw I was taking about in previous paragraph: To be able to see some content through the publisher portlets within the portal page, you can see from the above explanation that multiple calls to the publisher DB will be made. Let's say we have 5 publisher portlets on the page (not uncommon), for each page rendering for 1 user, 5 DB calls will be made to the publisher DB (in addition of multiple other DB calls for portal and analytics). If we are now talking about thousands of users, we are talking about too many DB calls to simply see some content that does not change often. This has an unnecessary impact on DB load of your infrastructure, and will increase the page load consequently since DB calls are inherently slower that simple content rendering...

    While reading, some of view are already thinking for a good reason: CACHING! Yeah, indeed, caching is the secret to scaling and performance (not always necessary to take out the big bucks and supercharge even more the DB infrastructure). And great for us, the portal offers a great out-of-the-box caching capability within the web service object: simply set the minimum caching to 2 hours, max to 20 days, and normally, you would think that the portal should simply cache the published content for that amount of time...removing the need to call the published_content_redirect.jsp altogether, and thus the need to make a DB call!! ...but it does not happen this way. You don't believe me? Enable access logging on publish content redirector components (un-comment "Access logger" section within <ALUI HOME>\ptcs\6.4\container\deploy\jbossweb-tomcat50.sar\server.xml) and you will clearly see that even though your page contains only published content portlets that should be cached, the published_content_redirect.jsp  is still constantly called...and thus caching is not really....caching.

    Why does this happen? It is because of the redirect mechanism for serving content...From www.w3.org, 302 is explained this way: "The requested resource resides temporarily under a different URI. Since the redirection might be altered on occasion, the client SHOULD continue to use the Request-URI for future requests. This response is only cacheable if indicated by a Cache-Control or Expires header field." Thus, the portal is doing is job perfectly (it acts as a client here) and will never cache a response with a temporary redirect 302 status code.

    Ok so what can make it better? Changing the redirect mechanism to something that does not redirect...Well, using this other JSP actually already present and developed within ALUI 6.4 (I have not verified if present in earlier versions of publisher): published_content_noredirect.jsp. Instead of issuing a redirect to the published content, this jsp performs an HTTPGetRequest to it and write its content to the JSP output stream. The response code is now a simple 200 OK ("The request has succeeded" - www.w3.org) that can be cached by the portal. To enable this, simply change the HTTP url of the Published Content Web Service Object to be published_content_noredirect.jsp instead of published_content_redirect.jsp. Of course, check out the publisher redirector access logs to see the dramatic difference...under load, you will initially see a bunch of requests to published_content_noredirect.jsp, but very very fast, the access log becomes silent, all the content being really cached by portal...

    Result?? You can increase the load even more, and the page response satys the same (or is even better), and the DB utilization is not altered by that load...you simply have a site that is so much more performant. Our initial results showed that under constant intensive load (with and without the change) the publisher infrastructure would not crash anymore, the CPU usage of both DB and publisher redirectors would be consequently diminished, the number of DB requests would drop, and the load could actually be increased to new levels without loss of performance and functionality, thanks to caching.

    Since this post is already long, I am going to stop here for now...Please read next post (http://fsanglier.blogspot.com/2008/06/alui-publisher-part-2-increase_30.html) to understand the second flaw: the published_content_noredirect.jsp has a truncation bug...that is fixable of course :)

    So long!

    Monday, January 28, 2008

    ALUI Administration Tool for Environment Refresh: String Replacing for URLS

    Any company that owns a portal infrastructure such as ALUI will usually also plan for extra portal environments for troubleshooting and test purposes. Any portlet developments, portal customizations, or administrative changes will be implemented and tested in those environments. This is very good practice indeed. But you will quickly notice something happening: Your production portal is no longer in sync with your other staging / QA / Test environments. Although not necessarily a huge deal in the immediate future, that could become a problem when your test environment is really too different from the production one: Imagine the test team trying to execute a test plan in Staging environment, but half the production communities are simply not present in the test environment...etc...etc...etc...

    I am sure you figured where I am going: Sooner or later, a clean refresh of your tests environments will be needed. Such a refresh is best done by a full database copy from production to your test environment (ALUI built-in export / import functionality is not suited for a large amount of objects, and thus is totally inappropriate and not recommended in the present case)   

    Besides restoring all the appropriate repositories (documents, published content, search indexes etc...), a major problem remains: All the various service' urls specified in the various ALUI objects (Remote servers, web services, KD cards, server urls, etc...) are no longer accurate in the refreshed test environments (of course: those urls are the production ones). Thus  your test environment is far from working yet.

    2 possibilities from here (1 good and 1 less good):

    • You use host files in your test environments. These hosts files will direct the same production urls to the appropriate test servers (not good in my point of view because it is dangerous: if the hosts files is removed, there is a huge risk of having your test environment accessing production - thus not good :) )
    • You use environment-specific DNS entries for each services. (Much better because no risk of inter-environment communication)

    First, it is much better practice to call each service through DNS name rather than directly to through the server name. And secondly, it makes it much easier when it comes to environment refresh.

    Why? Because all the DNS entries follow a global "intelligent" pattern (i.e. PUBLISHER.COMPANY.COM / PUBLISHER-TEST.COMPANY.COM / PUBLISHER-DEV.COMPANY.COM / etc...), it is much easier to create a tool (or a DB query for those DB gurus out there) that will allow for automatic replace of these DNS names.

    That's where I wanted to lead you:

    I created a tool that allow just that: for all the portal objects that contain URLs, the tool can replace a specific pattern with another one that you define. Although I could have created a set of DB queries that could do the job (I actually started to do that when I had to do my first environment refresh), I realized that a utility written in JAVA (most portable language) and using ALUI server API would guaranty portability, extensibility and reusability. The main advantage is that it is completely independent from backend technologies, and can work with any ALUI portal (.NET or JAVA), as well as any database (MS SQL or Oracle). It has been tested with JRE 1.4 and used by me on all G6 portal versions (until 6.1 MP1 included)

    DISCLAIMER: ALTHOUGH I PERSONNALY USE THIS TOOL, THERE IS NO GUARANTY; SO USE THIS TOOL AT YOUR OWN RISK blah blah blah AND USE IT ONLY IF YOU ARE PROFFICIENT ENOUGH WITH ALUI PORTAL TECHNOLOGIES.

    2 smart moves if you are not the easily scared type of person: First, make a DB backup of the test environment you are going to refresh, and second, test it in your local environment first, and see with your own eyes that it works perfectly well :)

    For the moment, this migration utility only updates the main portal objects that contain urls:
        -Remote servers
        -Web Services
        -Portal Settings

    When I have time, I'll improve the utility and include publisher content items (especially urls embedded in them) in the list of migration objects.

    Shell scripts (.bat and .sh) are created in order to facilitate and secure the usage of this utility. The user must provide several key information for the program to run:
        -Application name (by default: portal)
        -Administration username (must be an ALUI local database user)
        -Username password
        -Pattern to look for (any valid regular expression)
        -Replacement String
        -Optional: Debug mode? (if true, details of the parsed objects is output to screen)

    The executable JAR along the appropriate launch scripts (bat for windows, sh for unix/linux) must be copied into the <PT_HOME>/bin folder in order to access the required libraries (otherwise the shell script -especially the java classpath- can be changed appropriately)

    I zipped this tool (scripts and jar) for your convenience and uploaded it (pturlreplace-1.0.zip) to my google code project (http://code.google.com/p/alui-toolbox/).

    Let me know what you think of it after you tested it. If you think it is useful, I'll update this post with new version of the tool as I come up with it. Enjoy!

    Wednesday, December 5, 2007

    ALUI and Java Server Faces Portlet development - Part 2 (Continued)

    In my previous ALUI JSF post ALUI and Java Server Faces Portlet development - Part 1, I talked about the problems you might encounter while using JSF and ALUI, and I also started to explain how to fix those problems. I will continue with that post...

    Form Unique Id

    It is standard (best) practice to make sure every ids and every javascript function names in a HTML page are unique. That way, Javascript functions are targetted correctly when called, and Javascript functions can effectively use "getElementById()" to access the right element within the portlet. When you are in a Portal environment, it is difficult to ensure that all the portlets on the page have unique ids. And thus it is possible to have a defective portlet when rendered with others on the same page, due to those kind of interferences (same element ids or same javascript function names). The probably of error reach 100% of chance if you put 2 portlets with identical content on the page.

    Traditionally, with web applications where you have control over every elements and javascript function throughout the page (i.e. standard JSP), you would surely use the pt:common.namespace adaptive tag, in order to append to every JS functions and form ids the token string (i.e. $$PT_TOKEN$$)

    But unfortunately, that won't work with JSF. Why is that?

    First, quick reminder: It is important to note that "namespace" tag (like every ALUI Adaptive Tags) is transformed by the GATEWAY component. In other word, if you use the tag $$PT_TOKEN$$ throughout your JSF form, the remote JSF application DO see literally "$$PT_TOKEN$$", not the actual portlet IDs. When the portlet response goes through the gateway component, that is when the tag "$$PT_TOKEN$$" is changed on the fly with the real portlet id number.

    Since JSF is a component oriented framework that generates the HTML markup, we don't have direct control over the HTML elements...and also, the element "name" (this is posted back to the server) is usually the same as the element "id". Thus, when you use:

    <h:form id="editaddress_$$PT_TOKEN$$">
    <h:inputText id="fullname" value="#{bean_backing.fullname}" />
    </h:form>



    You HTML markup for the response is:



    <form id="editaddress_$$PT_TOKEN$$" ="">
    <input id="editaddress_$$PT_TOKEN$$:fullname"
    name="editaddress_$$PT_TOKEN$$:fullname" type="text" value="" />
    <form>



    And when you visualize the portlet source code (as rendered by the portal, where 289 is the portlet id that renders your web application), you get:



    <form id="editaddress_289" ="">
    <input id="editaddress_289:fullname"
    name="editaddress_289:fullname" type="text" value="" />
    <form>



    So great, we do have unique HTML elements within each portlets, but you start to get the problem, right? Okay I am still going to explain :)



    So the JSF component is waiting for a posted request attribute with literal name "street_$$PT_TOKEN$$", but actually, the posted attribute is "street_289"...and thus the JSF component cannot decode the posted value, and thus the form post won't work.



    So the solution is to include the real portlet Id on the remote application side directly. I will use the same technique that I explained in the previous post: Custom renderer for the Form JSF component, overriding this time the base method "convertClientId(FacesContext context, String clientId)". Within the overridden method, I get the portlet id by simply using the IDK and then append the id to the component id.



    @Override
        public String convertClientId(FacesContext context, String clientId) {
            IPortletContext ctx = PortletContextFactory.createPortletContext(
                    (HttpServletRequest)context.getExternalContext().getRequest(),
                    (HttpServletResponse)context.getExternalContext().getResponse()
                    );
            IPortletRequest portletRequest = ctx.getRequest();
            //add the portlet id to the form
            if(portletRequest.isGatewayed())
                clientId += ctx.getRequest().getPortletID();
            return super.convertClientId(context, clientId);
        }



    That's it!! That way, even if you let JSF generate the form element id, the "portlet id" WILL be appended if you are viewing the application through the portlet. And since the component is aware of the portlet id, the post will work fine. And finally, since the <h:form> is very often the top element of your JSF page, all the sub elements will also be unique since the form id is the prefix for all the children component ids.



    Although you probably don't want to create a custom specific renderer for every components you want to use, this technique seems the best to me because it is completely non-intrusive within your JSF page...and thus, in the future, if BEA corrects all those JSF problems, you just disable the custom renderers and everything will work without any change.

    Saturday, November 17, 2007

    BEA Aqualogic Portal (ALUI) - Single Sign On (SSO) Concerns and Techniques

    As explained very well on this great post "Changing SSO Settings" from Function1, it is highly recommended to protect only the "SSOServlet" (default path= /portal/SSOServlet) when you want to SSO-enable the ALUI Portal. As I worked recently with an integration of ALUI with CA Siteminder, I am going to go a little bit further so that the SSO solution is complete.

    But first, the 2 benefits of protecting only SSOServlet are:

    • Performance (as explained in Function1's post)
    • Enable guest access to the portal with same url (the portal is the gate, not the SSO agent)

    But the main concerns are related to the SSO Session versus the Portal Session. As explained in Function1's post, the timeouts of portal and SSO agent should be in sync so that the user is actually logged out when the portal session times out (20 minutes by default). As it is well explained, you want both timeout to be very close, the portal timeout being trigged a bit before the SSO timeout.

    But that's where I want to go just a little further...timeout sync is not enough to make the system perfect. Indeed, 2 extra use cases exist (and are not covered yet):

    1. SSO session "keep-alive"
    2. SSO session Logged out.

    SSO session "keep-alive"

    Basically, after the user successfully logged into ALUI portal (using SSO login screen), a SSO token has been created for its session (SSO cookie embedded in the browser session). But the problem is that after a successful login, the portal won't call SSOServlet again, and thus the SSO agent (generally on the web server front end, or the application server hosting the portal) has no longer anything to say about the traffic going through its agent since it is all unprotected from its point of view. Actually, it is as if the agent is not even seeing that traffic at all. Why do we care?? Well, the SSO session timeout is only refreshed if the agent is actually seeing any protected traffic. When the agent is not seeing any protected traffic, the timeout count down starts.

    So to sum up, when the user successfully logged in the portal and is browsing the portal protected content, it is as if the session was idle from the Siteminder point of view. That is perfectly fine (at least you wont notice the problem) if you only use SSO for the portal. But imagine you use SSO for any other application in your enterprise (which is the use of SSO by the way :) ), or even portlets within your ALUI portal, then you probably don't want your SSO session to timeout while you are browsing the portal. Otherwise, after 20 or 30 minutes (whatever is set up on your SSO system for SSO timeout), you will still be able to browse the portal fine (since you are not idle from the portal point of view), but your SSO Session WILL HAVE TIMED OUT...and you will have to login again if you go to another of your favorite enterprise SSO protected sites...

    So how can we make that work?? How can we ensure that the SSO session (or SSO token) is "kept alive" while we browse the portal?? Very simple trick actually: I basically created a dummy SSO protected content on the front end web server (for example http://portal.com/sec/dummy.html) and embedded that url in an invisible iframe in every portal pages (well suited for the footer portlet of your portal, or better, put that in your favorite liquidskin component - see my liquidskin ongoing serie "Overview of BEA ALUI LiquidSkin: Part 1" and "Overview of BEA ALUI LiquidSkin: Part 2")...

    Basically, you have <iframe src="http://portal.com/sec/dummy.html" width="0" height="0"></iframe> at the bottom of every page you wish...it seems low tech, but works pretty well: every portal pages that have that footer portlet will initiate a request to that protected url, letting the SSO agent know: Hey, i am not idle, my session is still alive!!

    Ok, good...but I am still not happy yet :)

    SSO Log out

    When you logout from the portal, you also want the SSO session to be logged out too. this is easy enough with CA siteminder: in the SSO policy you just provide the log out pattern to look for (usually something like /portal/server.pt?open=space&name=Login&control=Login&doLogout=&clearsession=true">/portal/server.pt?open=space&name=Login&control=Login&doLogout=&clearsession=true">/portal/server.pt?open=space&name=Login&control=Login&doLogout=&clearsession=true">http://<portal>/portal/server.pt?open=space&name=Login&control=Login&doLogout=&clearsession=true) and the SSO agent (remember it is on your front end web server or application server) will kill the SSO token if it intercept such a pattern in the request url. Cool!! That way, when you click log off in the portal, you are actually logged out.

    But again, let's imagine we have multiple systems that uses the SSO session. You first loggedin the portal, browse it a little, and then decide to go to this other SSO-protected application. Then you decide you are finished and logout from there. What happens if you log out from another system apart from the portal?? Well if you don't do anything special, you will logoff from this other application, which will normally kill the SSO session (same "finding logout url pattern" technique)...but if you go back to the portal after that (and you were previously logged in)...oh surprise, you can go in no problem!! Well that's normal, your portal session is not over yet (provided that you were not away from the portal more than the portal timeout) and you did not do anything to finish it...

    But that is not how it should work, right? If the SSO session is over (by logging off from any SSO protected system), you should not be able to go in any other SSO protected system!! Just imagine if your favorite banking web site was working this way...you logoff from your savings account, but your checking account interface is not logged off...I would not particularly like that from a security point of view :)

    So we have to deal with it: I see 2 options in order to make sure all your systems are logging out when 1 is logging out:

    • Having an independent logout page that initiates logout requests for all the protected applications in your enterprise (for example an HTML page containing as many <img> tags or <iframe> tags as you have application to log out) -- I don't like this method so much because I will have to maintain this central logout page as soon as I add / remove a protected application in my environment,
    • Making applications log out on their own if the SSO token is not there. (or at least forcing any requests to SSO login if the SSO token is not present in the request)

    The second solution is to me the best since it works once and for all. But it is a little more complex than the first one since we are protecting only the SSOServlet...(if the site was fully protected, that would work without doing anything...the agent would not see the SSO token and would not let you pass as long as you don't authenticate successfully)

    How can we do that in the portal? I chose to implement a JAVA filter (I was working with JAVA portal with that client) that checks for SSO token presence (you can implement the same filter functionality in DOTNET too: it is called http module). The simplified logic is:

    • If SSO token is present and valid, do nothing,
    • Otherwise, force portal logout (if you actually previously logged in)...which will make the request being redirected to SSO login...

    Et Voila! That way, the portal will logout nicely, on his own, when no SSO token is found.

    I hope this helps you better understand the different concerns of implementing Single Sign On with your favorite ALUI portal (would actually apply to other portals too)

    Saturday, November 3, 2007

    How to Change ALUI Components Passwords

    Each ALUI component requires Database information (host, schema, user, password, etc…) in order to interact with their respective Database.

    When DBAs change the password of a database user (in some companies, it can actually happen pretty often) that is used by an Aqualogic User Interaction component, a similar “password change” procedure has to be done for each the concerned components’ configuration files.

    If that was just it, that could be easy...but thanks to security, every passwords in the ALUI configuration files are encrypted...and thus, you need to encrypt the password too. But how do you encrypt those passwords? Where is the encryption utility?

    ALUI provides several encryption mechanisms that you can apply to each component. I will try to walk you through what I call the encryption maze so that changing ALUI passwords becomes a breeze. (Note: files *.bat or *.cmd exist also in the unix/linux world...extension sh)

    ALI Portal

    DB Configuration File(s) path :

    <PORTAL_HOME>\settings\common\serverconfig.xml

    Encryption Utility path:

    • <PORTAL_HOME>\ptportal\<VERSION>\bin\ptconfig.exe (this utility will change the password directly in the serverconfig.xml file, OR
    • <PORTAL_HOME>\ptportal\<VERSION>\bin\cryptoutil.bat (must be executed in a shell or dos prompt -- provides the encrypted version of the text provided)

    ALI publisher / Workflow

    DB Configuration File(s) path:

    • <PORTAL_HOME>\ptcs\<VERSION>\settings\config\database.content.properties (publisher DB connection)
    • <PORTAL_HOME>\ptcs\<VERSION>\settings\config\database.portal.properties (portal DB connection)
    • <PORTAL_HOME>\ptworkflow\<VERSION>\settings\config\application.conf (workflow DB connection)

    Encryption Utility path:

    <PORTAL_HOME>\ptcs\<VERSION>\bin\native\pcsencrypt.cmd

    Note: You can use that same utility to encrypt the publisher basic authentication user password too.

    ALI Collaboration

    DB Configuration File(s) path:

    • <PORTAL_HOME>\ptcollab\<VERSION>\settings\config\database.xml (collaboration DB connection)
    • <PORTAL_HOME>\ptnotification\<VERSION>\settings\config\database.xml (notification DB connection)

    Encryption Utility path:

    <PORTAL_HOME>\ptcollab\<VERSION>\bin\passwordChanger.bat

    The utility passwordChanger.bat will change the password in both config file and database directly. This is very useful indeed, but what if the DBA only can change the password, or what if he already did change it...What now? Well another solution is available to you...you can use another encryption utility that works (weirdly): The publisher encryption utility <PORTAL_HOME>\ptcs\<VERSION>\bin\native\pcsencrypt.cmd

    Note: You can use that same utility to encrypt the collaboration basic authentication user password too.

    Studio

    DB Configuration File(s) path:

    • <PORTAL_HOME>\ptstudio\<VERSION>\settings\config\PTStudioConfig.xml (studio and portal DB connections)

    Encryption Utility path: No encryption utility is provided in the studio directory, so believe it or not...you guessed it...again the publisher encryption utility can be used for both studio and portal DB passwords.

    <PORTAL_HOME>\ptcs\<VERSION>\bin\native\pcsencrypt.cmd

    Analytics

    • <PORTAL_HOME>\ptanalytics\<VERSION>\settings\config\security\securityservice-config.xml (portal DB connection)
    • <PORTAL_HOME>\ptanalytics\<VERSION>\settings\config\hibernate.properties (analytics DB connection)
    • <PORTAL_HOME>\ptanalytics\<VERSION>\settings\config\configurator.properties (analytics, portal, collaboration and publisher DB connections)
    • <PORTAL_HOME>\ptanalytics\<VERSION>\settings\config\jobs.properties (portal, collaboration and publisher DB connections)

    Encryption Utility path: Analytics provide a web configuration interface for changing DB connections and passwords in all the configuration files. But if you need to do it yourself without this web interface, again a solution exist: this time use the ... portal encryption for all the passwords!!

    <PORTAL_HOME>\ptportal\<VERSION>\bin\cryptoutil.bat

     

    That's all I got for now! Hope this helps to navigate in ALUI Password encryption hell! :)