New Portal Tools On ALUI Toolbox

In my efforts of improving the ALUI/Webcenter portal, and especially enhancing its admin management capabilities, I created over time a set of utilities that I think could be useful to the ALUI/Webcenter community. In one of my previous post, I already talked about the "PT URL Replace" utility (refer to: http://fsanglier.blogspot.com/2008/01/alui-administration-tool-for.html) which was already on the ALUI Toolbox Google code project as a download only. What I did some days ago is updating the ALUI Toolbox Google project (http://code.google.com/p/alui-toolbox/) with the "PT URL Replace" code, as well as a couple of new applications/utilities:

• Portlet Caching Clearer (web portlet - c#)
• Object Identifier (web portlet - c#)
• Web Service Changer (web portlet - c#)
• Page Lister (web portlet - c#)
• ALUI Knowledge Directory Security Agent (java app runnable from console and/or scheduled task such as ALUI jobs) – feature download at http://code.google.com/p/alui-toolbox/
• Improvements for PT URL Replace utility – http://code.google.com/p/alui-toolbox/

All the above apps use the Server API (for Server API introduction, refer to previous posts:) because the tasks performed would not be possible by simply using the IDK. It has been written for ALUI 6.1.x versions and might not work fully on version 6.5 and above without minor changes (because the server API does change between portal releases). When I get the chance, I will update the Google code project with 2 extra branches that follows the more current portal versions.

The code is released under the GPL license (you can find a copy the the GPL license in the root folder of the apps, or go to http://www.gnu.org/licenses/gpl.html), and off course is provided without any kind of warranty...

I'll go quickly over each of these utilities in order for you to understand why they might be useful.

Portlet Caching Clearer:
In order to maximize performance, each portlet in the portal can have output caching enabled to a particular timeframe. The drawback of such caching mechanism is that the updates performed by content managers are not instantaneously viewable to end users. This portlet that can be added on your "My Page" allows you to clear portlet caching in 3 different ways:

1. Using the portal tree picker, select the portlet(s) you want to clear.
2. Using the portal tree picker, select the "Community Page(s)" whose portlets you want to clear. The utility will find all the portlets currently added to the selected pages, and clear the cache for each of them.
3. Using the portal tree picker, select the "Community(ies)" you want to clear. The utility will find all the portlets currently added to the selected pages of the selected communities, and clear the cache for each of them.

That way the content managers can easily clear caching for a set of pages, communities, or portlets...

Object Identifier:

Let's say that you have a portlet application that identify a portal object in its config file by its UUID (or its ID for that matter)...Now let's say you come back to that portlet 6 month later because you need to perform an improvement and/or fix something...Unless you clearly documented what object corresponds to this ID/UUID (and where it is in the portal), it will not be easy to find it (unless you can easily run a DB query...). This portlet basically answer that needs:

• Provide a UUID and it will tell you the corresponding Classid/ObjectID pair + Object Name + Object Location in the Portal Admin Hierarchy.
• Provide a Classid/ObjectID pair, and it will tell you the corresponding UUID + Object Name + Object Location in the Portal Admin Hierarchy.
  

Web Service Changer:

Have you ever noticed that you cannot change the webservice attached to a portlet once you initially picked it and created the portlet? Now let's say that you have a bunch of publisher portlets that are all tied to the same "Publish Content Web Service" object. But all a sudden, you change your mind and decide to have some portlets that should be tied to 2 different "Publish Content Web Service" objects, each one with a different caching timeframe (i.e. a long caching for the content that hardly change, and a shorter caching for the content that changes often). In the portal out of the box, you cannot do that easily, and will probably have to recreate all the portlets and re-attach them to the same publisher content etc...(big pain).

Well this portlet allows you to easily change the "portlet web service" OR the "portlet template" attached to a particular portlet or group of portlets:

• Using the portal tree picker, pick the portlet you want to change.
• Using the portal tree picker, pick the web service OR portlet template that you want these portlet to be assigned to.
• Click Submit...Done.

Page Lister:

This simple utility allows to display as a list of HTML links (simple <a href=""></a> tags) all the Community Pages in the portal that a certain users have access to...

Why doing this? it is a simple way of creating a Sitemap that a web "crawler" (either ALUI web crawler, or Google appliance etc...) could hit in order to be dynamically aware of all the ALUI pages of your site that are accessible to the guest users for example...Or that could also be used as a security monitoring tool in order to verify which pages are accessible to a particular user (i.e. Guest) and ensure it is not a security mistake etc...

ALUI Knowledge Directory Security Agent:

This utility, written in java (we have to think about our linux/unix user base too :) ), was created as a scheduled task agent in order to act as a security cop in the knowledge directory. What it does is go through all the KD folders the agent user has access to, and automatically assign the found cards with their parent folder security.

All you need to provide are:

1. KD folder ID to start with,
2. A user ID / password (or session token if you run as a portal "external operation") the agent should impersonate with,
3. (Optional) CrawlerIDs (if you want to change only the cards that were brought into KD by a specific set of crawlers)

2 main use cases I see for this tool:

• Use it as a "cop" background schedule job in order to ensure the security on the cards is always right, based on the security of the folder.
• In the event you use a single crawler with various filters that organize your content in various folders. Without this agent, the crawled cards will get the security defined in the crawler's "crawled content permission" section...and that might not be in phase with the security of the various KD folders the content will be organized into (due to filters). If you run this cop security agent after each crawl, then you are all set based on the destination folder, not on the crawler's "crawled content permission" section...

That's it for now. I hope you'll test them out and let me know if you think these are useful or not.
Don’t hesitate to give ideas and/or wish lists of things which would be good to have in the portal...and don't hesitate to share the cool utilities you've done too

I or others will be updating the project every so often so keep informed or you might miss out on some good resources :)

ALUI Tool: URL (or text) Migration within Publisher Items

Following my previous article "ALUI Administration Tool for Environment Refresh: String Replacing for URLS" talking about migration between environments, here is an extra piece that you might find very useful (I surely use it all the time)

Basically, as explained in the previous article, it is common to have different DNS aliases set up per environment...I.e for publisher remote server, you could have:

• http://publisher.domain.com for production
• http://publisher-stg.domain.com for staging
• http://publisher-dev.domain.com for development

Similarly, the publish browsing URL is not an exception to this rule:

http://publisher-content.domain.com/publish  for production

http://publisher-content-stg.domain.com/publish for staging

http://publisher-content-dev.domain.com/publish for development

When you add an image or a link in the free text editor of content items in publisher, it will most of the time create an absolute URL to that resource...thus you can imagine that there will be a lot of DNS aliases within a lot of publisher items throughout the environment.

What happen when you migrate the publisher DB from one environment to another? Well you will have a lot of DEV dns aliases within your Staging environment (in case of a DEV promotion to Stage); or a lot of production DNS aliases within your dev environment in the case of a production refresh to DEV.

In my previous article "ALUI Administration Tool for Environment Refresh: String Replacing for URLS", I was mostly talking about migrating URL within portal objects, but nothing really about migrating urls within publisher items.

Thus, I created some DB scripts (SQL Server only for now) that do just that...

1. puburls-PTCSDIRECTORY-nvarchar-replace.sql: Script to change a particular string within the PUBLISHEDTRANSFERURL and PUBLISHEDURL columns (which is mapped out in the DB to a column of type VARCHAR)
2. puburls-PTCSVALUE-ntext-replace.sql and puburls-PTCSVALUEA-ntext-replace.sql: Scripts to change a particular string within the "long text" property of a publisher item (which is mapped out in the DB to a column of type TEXT)
1. PCSVALUES.LONGVALUE (hosting the long text of the currently published item)
2. PCSVALUESA.LONGVALUE (hosting the long text values of all the previous versions of the item)

For the first script, PUBLISHEDTRANSFERURL and PUBLISHEDURL columns are of type VARCHAR and thus it is easy to replace a string within those columns using the REPLACE MS SQL Function. Thus, a simple SQL statement is good here.

The main challenge was really with the 2nd scripts...indeed, within a column of type TEXT, the SQL "REPLACE" function cannot be used...The workaround is to use the PATINDEX and UPDATETEXT functions within a Transact-SQL (T-SQL) script. To give the credit to to the right person, I adapted a script that I found at ASP FAQ - How do I handle REPLACE() within an NTEXT column in SQL Server?

DISCLAIMER: ALTHOUGH I PERSONNALY USE THIS SCRIPT ALL THE TIME, THERE IS NO GUARANTY; SO USE THIS TOOL AT YOUR OWN RISK blah blah blah AND USE IT ONLY IF YOU ARE PROFFICIENT ENOUGH WITH ALUI PORTAL TECHNOLOGIES.

Attached is the zip file package that contains the 3 scripts:

• puburls-PTCSDIRECTORY-nvarchar-replace.sql
• puburls-PTCSVALUE-ntext-replace.sql
• puburls-PTCSVALUEA-ntext-replace.sql

Don't forget to change the string to look for, and the string to replace it with

puburls-PTCSDIRECTORY-nvarchar-replace.sql

UPDATE [dbo].[PCSDIRECTORY]
SET 
[PUBLISHEDTRANSFERURL]=REPLACE([PUBLISHEDTRANSFERURL],'-DEV.DOMAIN.COM','-TST.DOMAIN.COM'), 
[PUBLISHEDURL]=REPLACE([PUBLISHEDURL],'-DEV.DOMAIN.COM','-TST.DOMAIN.COM')
WHERE 
publishedtransferurl like '%-DEV.DOMAIN.COM%' 
or publishedurl like '%-DEV.DOMAIN.COM%'

puburls-PTCSVALUE-ntext-replace.sql and puburls-PTCSVALUEA-ntext-replace.sql

SET @oldString = N'por-pubcontent-dev.domain.com'; -- remove N 
SET @newString = N'por-pubcontent-tst.domain.com'; -- remove N 

That's it! Let me know if you find it as useful as I do! Enjoy!!

ALUI Administration Tool for Environment Refresh: String Replacing for URLS

Any company that owns a portal infrastructure such as ALUI will usually also plan for extra portal environments for troubleshooting and test purposes. Any portlet developments, portal customizations, or administrative changes will be implemented and tested in those environments. This is very good practice indeed. But you will quickly notice something happening: Your production portal is no longer in sync with your other staging / QA / Test environments. Although not necessarily a huge deal in the immediate future, that could become a problem when your test environment is really too different from the production one: Imagine the test team trying to execute a test plan in Staging environment, but half the production communities are simply not present in the test environment...etc...etc...etc...

I am sure you figured where I am going: Sooner or later, a clean refresh of your tests environments will be needed. Such a refresh is best done by a full database copy from production to your test environment (ALUI built-in export / import functionality is not suited for a large amount of objects, and thus is totally inappropriate and not recommended in the present case)   

Besides restoring all the appropriate repositories (documents, published content, search indexes etc...), a major problem remains: All the various service' urls specified in the various ALUI objects (Remote servers, web services, KD cards, server urls, etc...) are no longer accurate in the refreshed test environments (of course: those urls are the production ones). Thus  your test environment is far from working yet.

2 possibilities from here (1 good and 1 less good):

• You use host files in your test environments. These hosts files will direct the same production urls to the appropriate test servers (not good in my point of view because it is dangerous: if the hosts files is removed, there is a huge risk of having your test environment accessing production - thus not good :) )
• You use environment-specific DNS entries for each services. (Much better because no risk of inter-environment communication)

First, it is much better practice to call each service through DNS name rather than directly to through the server name. And secondly, it makes it much easier when it comes to environment refresh.

Why? Because all the DNS entries follow a global "intelligent" pattern (i.e. PUBLISHER.COMPANY.COM / PUBLISHER-TEST.COMPANY.COM / PUBLISHER-DEV.COMPANY.COM / etc...), it is much easier to create a tool (or a DB query for those DB gurus out there) that will allow for automatic replace of these DNS names.

That's where I wanted to lead you:

I created a tool that allow just that: for all the portal objects that contain URLs, the tool can replace a specific pattern with another one that you define. Although I could have created a set of DB queries that could do the job (I actually started to do that when I had to do my first environment refresh), I realized that a utility written in JAVA (most portable language) and using ALUI server API would guaranty portability, extensibility and reusability. The main advantage is that it is completely independent from backend technologies, and can work with any ALUI portal (.NET or JAVA), as well as any database (MS SQL or Oracle). It has been tested with JRE 1.4 and used by me on all G6 portal versions (until 6.1 MP1 included)

DISCLAIMER: ALTHOUGH I PERSONNALY USE THIS TOOL, THERE IS NO GUARANTY; SO USE THIS TOOL AT YOUR OWN RISK blah blah blah AND USE IT ONLY IF YOU ARE PROFFICIENT ENOUGH WITH ALUI PORTAL TECHNOLOGIES.

2 smart moves if you are not the easily scared type of person: First, make a DB backup of the test environment you are going to refresh, and second, test it in your local environment first, and see with your own eyes that it works perfectly well :)

For the moment, this migration utility only updates the main portal objects that contain urls:
    -Remote servers
    -Web Services
    -Portal Settings

When I have time, I'll improve the utility and include publisher content items (especially urls embedded in them) in the list of migration objects.

Shell scripts (.bat and .sh) are created in order to facilitate and secure the usage of this utility. The user must provide several key information for the program to run:
    -Application name (by default: portal)
    -Administration username (must be an ALUI local database user)
    -Username password
    -Pattern to look for (any valid regular expression)
    -Replacement String
    -Optional: Debug mode? (if true, details of the parsed objects is output to screen)

The executable JAR along the appropriate launch scripts (bat for windows, sh for unix/linux) must be copied into the <PT_HOME>/bin folder in order to access the required libraries (otherwise the shell script -especially the java classpath- can be changed appropriately)

I zipped this tool (scripts and jar) for your convenience and uploaded it (pturlreplace-1.0.zip) to my google code project (http://code.google.com/p/alui-toolbox/).

Let me know what you think of it after you tested it. If you think it is useful, I'll update this post with new version of the tool as I come up with it. Enjoy!

Technorati Tags: ALUI Portal,Portal Maintenance,Technical Tips,Tools

ALUI Portlet Monitoring Tool

Most ALUI Portal environments will usually contain a large number of portlets, either "out-of-the-box" or custom portlets. These portlets, which range from simple bookmark links to sophisticated applications that interact with third party products such as Siebel or SAP, are indeed the major components that serve the high value functionality of your ALUI portal. Thus, it is vital that these portlets are monitored properly, so that a problem can be detected as soon as possible (preferably before your customer calls you to report the error that you had no idea was occurring...) and of course corrected asap.

Developed by Project Performance Corporation (www.ppc.com), Portlet Monitor® is a utility designed to track the operational status of all portlets registered in any communities or mypages of your ALUI Portal environment. Although Rakesh Gupta from PPC is the creator of this product, I personally redesigned and upgraded it for version 6.X of ALUI, and thus know the internals of it inside out.

You can contact me (fsanglier-at-ppc-dot-com) if you need additional information and pricing.

What is it?

Portlet monitor is a stand alone application (developed in .NET) that uses the Port

al Server API to read the portal data, especially the community and portlet objects. It can be installed on any server that have the following components: Portal, Automation, or API Service (to be able to use the server API). Using windows automated tasks (or even ALUI automation server) portlet monitor can be configured to run at any pre-specified time.

How does it work?

In plain English, Portlet Monitor connect as a portal user (by initiating a portal session) and iterate through all communities, mypages and finally portlets (the ones that are actually used on the community or "My" pages) that this user has access to. For each of the portlet found, Portlet monitor will "fake" a gateway call to the remote server hosting the portlet, passing all the required portal preferences and other ALUI-specific settings to the remote portlet application.

Error Found?

Various types of error can be caught by portlet monitor:

• Various HTTP Errors (500 / 401 / etc...) returned by the remote portlet application/web server.
• Portlet Timeout errors
• Network errors (remote portlet server has some network problems)
• Portal exceptions (portal settings or portlet preferences missing, problem with gateway component or any other portal internal component, etc...)
• Specific error text pattern found in the portlet (if the above errors are not found, an error can be triggered if a specific text patterns is found in a specific portlet)

Results?

The result of each run is locally saved in an XML file that contains the following:

• The portlet ID
• The community / mypage of the analyzed portlet
• Processing Time
• If "Broken", the last date/time the portlet was found "Working"
• If "Broken", the cause of the error (exception trace or error code)

After each run, an email containing the above XML report is automatically sent to the specified email list. (usually portal administrator and specific portlet developer(s))

It is also possible to eventually serve the XML report directly in the portal through a portlet (with a XSLT applied of course)

Performance?

Portlet monitor can be installed on any windows server that contains either the portal, automation or API service component (It is important to note that the portal component service does not need to be running) Thus, to minimize the performance impact of portlet monitor successive runs, it is recommended to install it on a server that is not part of the live portal cluster.

Why using this Tool versus Enterprise Monitoring Tools?

While most enterprise monitoring tools (i.e. NETIQ) work from the component service (or process) point of view, Portlet Monitor works from the portal point of view. What does it mean exactly?

Basically, enterprise monitoring tools are able to to monitor windows services, and eventually restart them if something wrong is happening. They are also able to monitor component log files and search for any text pattern that are relevant to an error, and if found, perform a specific action. (like restart the service) They are also able to perform a lot of other various operations...

But what they are not able to do in a portal context like ALUI (versus Portlet Monitor) are:

• Detect the errors before the end-user finds it (by running the portlet monitor scans at scheduled intervals) - compared to finding the errors in the log files when the end-user actually triggered the error himself...
• Detect point of failure(s) between the portal and the portlet remote server
• Detect portal application or database defects (i.e. gateway problem) that could induce the portlet in error (from portal point of view), even if the remote application is just working fine.
• Detect misconfiguration of the portlet that could induce the portlet in error (from portal point of view), even if the remote application is just working fine.
• Detect any infrastructure or software errors that would induce a timeout of the portlet.
• Detect any portlet errors even if the monitoring agent is not running (sometimes the monitoring agent can have defect too...)
• Detect any portlet errors even if the portlet application service is in a "Frozen" state (meaning the service is still up, the log are not showing any error, and thus the monitoring system is not reporting anything wrong, but the application is actually not responsive anymore)

It is important to note that portlet monitor is just a monitoring and notifying tool: It does not perform any action (like restarting services etc...) other than notifying the right people.

Last Words

If you don't have a monitoring infrastructure in place, this tool will offer you monitor effectively the most important and valuable components of your portal (portlets).

If you have already a monitoring infrastructure in place, this tool is not meant to replace it, but rather to be a valuable addition that completes your monitoring infrastructure already in place.

Technorati Tags: Aqualogic User Interaction,ALUI Portal,Tools