Thursday, June 14, 2012

Speed Reading

http://www.fourhourworkweek.com/blog/2009/07/30/speed-reading-and-accelerated-learning/

How much more could you get done if you completed all of your required reading in 1/3 or 1/5 the time?
Increasing reading speed is a process of controlling fine motor movement—period.
This post is a condensed overview of principles I taught to undergraduates at Princeton University in 1998 at a seminar called the “PX Project”. The below was written several years ago, so it’s worded like Ivy-Leaguer pompous-ass prose, but the results are substantial. In fact, while on an airplane in China two weeks ago, I helped Glenn McElhose increase his reading speed 34% in less than 5 minutes.
I have never seen the method fail. Here’s how it works…

The PX Project

The PX Project, a single 3-hour cognitive experiment, produced an average increase in reading speed of 386%.
It was tested with speakers of five languages, and even dyslexics were conditioned to read technical material at more than 3,000 words-per-minute (wpm), or 10 pages per minute. One page every 6 seconds. By comparison, the average reading speed in the US is 200-300 wpm (1/2 to 1 page per minute), with the top 1% of the population reading over 400 wpm…
If you understand several basic principles of the human visual system, you can eliminate inefficiencies and increase speed while improving retention.
To perform the exercises in this post and see the results, you will need: a book of 200+ pages that can lay flat when open, a pen, and a timer (a stop watch with alarm or kitchen timer is ideal). You should complete the 20 minutes of exercises in one session.
First, several definitions and distinctions specific to the reading process:
A) Synopsis: You must minimize the number and duration of fixations per line to increase speed.
You do not read in a straight line, but rather in a sequence of saccadic movements (jumps). Each of these saccades ends with a fixation, or a temporary snapshot of the text within you focus area (approx. the size of a quarter at 8? from reading surface). Each fixation will last ¼ to ½ seconds in the untrained subject. To demonstrate this, close one eye, place a fingertip on top of that eyelid, and then slowly scan a straight horizontal line with your other eye-you will feel distinct and separate movements and periods of fixation.
B) Synopsis: You must eliminate regression and back-skipping to increase speed.
The untrained subject engages in regression (conscious rereading) and back-skipping (subconscious rereading via misplacement of fixation) for up to 30% of total reading time.

C) Synopsis: You must use conditioning drills to increase horizontal peripheral vision span and the number of words registered per fixation.
Untrained subjects use central focus but not horizontal peripheral vision span during reading, foregoing up to 50% of their words per fixation (the number of words that can be perceived and “read” in each fixation).

The Protocol

You will 1) learn technique, 2) learn to apply techniques with speed through conditioning, then 3) learn to test yourself with reading for comprehension.
These are separate, and your adaptation to the sequencing depends on keeping them separate. Do not worry about comprehension if you are learning to apply a motor skill with speed, for example. The adaptive sequence is: technique ‘ technique with speed ‘ comprehensive reading testing.
As a general rule, you will need to practice technique at 3x the speed of your ultimate target reading speed. Thus, if you currently read at 300 wpm and your target reading speed is 900 wpm, you will need to practice technique at 1,800 words-per-minute, or 6 pages per minute (10 seconds per page).
We will cover two main techniques in this introduction:
1) Trackers and Pacers (to address A and B above)
2) Perceptual Expansion (to address C)

First – Determining Baseline

To determine your current reading speed, take your practice book (which should lay flat when open on a table) and count the number of words in 5 lines. Divide this number of words by 5, and you have your average number of words-per-line.
Example: 62 words/5 lines = 12.4, which you round to 12 words-per-line
Next, count the number of text lines on 5 pages and divide by 5 to arrive at the average number of lines per page. Multiply this by average number of words-per-line, and you have your average number of words per page.
Example: 154 lines/5 pages = 30.8, rounded to 31 lines per page x 12 words-per-line = 372 words per page
Mark your first line and read with a timer for 1 minute exactly-do not read faster than normal, and read for comprehension. After exactly one minute, multiply the number of lines by your average words-per-line to determine your current words-per-minute (wpm) rate.

Second – Trackers and Pacers

Regression, back-skipping, and the duration of fixations can be minimized by using a tracker and pacer. To illustrate the importance of a tracker-did you use a pen or finger when counting the number of words or lines in above baseline calculations? If you did, it was for the purpose of tracking-using a visual aid to guide fixation efficiency and accuracy. Nowhere is this more relevant than in conditioning reading speed by eliminating such inefficiencies.
For the purposes of this article, we will use a pen. Holding the pen in your dominant hand, you will underline each line (with the cap on), keeping your eye fixation above the tip of the pen. This will not only serve as a tracker, but it will also serve as a pacer for maintaining consistent speed and decreasing fixation duration. You may hold it as you would when writing, but it is recommended that you hold it under your hand, flat against the page.
1) Technique (2 minutes):
Practice using the pen as a tracker and pacer. Underline each line, focusing above the tip of the pen. DO NOT CONCERN YOURSELF WITH COMPREHENSION. Keep each line to a maximum of 1 second, and increase the speed with each subsequent page. Read, but under no circumstances should you take longer than 1 second per line.
2) Speed (3 minutes):
Repeat the technique, keeping each line to no more than ½ second (2 lines for a single “one-one-thousand”). Some will comprehend nothing, which is to be expected. Maintain speed and technique-you are conditioning your perceptual reflexes, and this is a speed exercise designed to facilitate adaptations in your system. Do not decrease speed. ½ second per line for 3 minutes; focus above the pen and concentrate on technique with speed. Focus on the exercise, and do not daydream.

Third – Perceptual Expansion

If you focus on the center of your computer screen (focus relating to the focal area of the fovea in within the eye), you can still perceive and register the sides of the screen. Training peripheral vision to register more effectively can increase reading speed over 300%. Untrained readers use up to ½ of their peripheral field on margins by moving from 1st word to last, spending 25-50% of their time “reading” margins with no content.
To illustrate, let us take the hypothetical one line: “Once upon a time, students enjoyed reading four hours a day.” If you were able to begin your reading at “time” and finish the line at “four”, you would eliminate 6 of 11 words, more than doubling your reading speed. This concept is easy to implement and combine with the tracking and pacing you’ve already practiced.
1) Technique (1 minute):
Use the pen to track and pace at a consistent speed of one line per second. Begin 1 word in from the first word of each line, and end 1 word in from the last word.
DO NOT CONCERN YOURSELF WITH COMPREHENSION. Keep each line to a maximum of 1 second, and increase the speed with each subsequent page. Read, but under no circumstances should you take longer than 1 second per line.
2) Technique (1 minute):

Use the pen to track and pace at a consistent speed of one line per second. Begin 2 words in from the first word of each line, and end 2 words in from the last word.
3) Speed (3 minutes):
Begin at least 3 words in from the first word of each line, and end 3 words in from the last word. Repeat the technique, keeping each line to no more than ½ second (2 lines for a single “one-one-thousand”).
Some will comprehend nothing, which is to be expected. Maintain speed and technique-you are conditioning your perceptual reflexes, and this is a speed exercise designed to facilitate adaptations in your system. Do not decrease speed. ½ second per line for 3 minutes; focus above the pen and concentrate on technique with speed. Focus on the exercise, and do not daydream.

Fourth – Calculate New WPM Reading Speed

Mark your first line and read with a timer for 1 minute exactly- Read at your fastest comprehension rate. Multiply the number of lines by your previously determined average words-per-line to get determine your new words-per-minute (wpm) rate.
Congratulations on completing your cursory overview of some of the techniques that can be used to accelerate human cognition (defined as the processing and use of information).
Final recommendations: If used for study, it is recommended that you not read 3 assignments in the time it would take you to read one, but rather, read the same assignment 3 times for exposure and recall improvement, depending on relevancy to testing.
Happy trails, page blazers.

Clean up spaces in DS

Cleaning up the spaces in DataStage


Clearing Lookup Table files
Problem
When a DataStage job with a lookup stage aborts, there may be lookuptable files left in the resource directories and they will consume space. The filenames are similar to "lookuptable.20091210.513biba"
Cause
When a job aborts it leaves the temporary files for postmortem review in the resource directories. Usually that is done in scratch, however, for lookup files, they are created in resource. Lookup filesets will not go away, just like regular datasets.
A lookup fileset looks like:
/opt/IBM/InformationServer/Server/Datasets/export.dsadm.abcdefg.P000000_F0000

A lookup file looks like:
/opt/IBM/InformationServer/Server/Datasets/lookuptable.20091210.513biba
Diagnosing the problem
Look for files with filenames similar to "lookuptable.yyyymmdd.nnnnnnn" left on disk when no jobs are running.
Resolving the problem
All files with lookuptable at the beginning of the filename can be removed as long as there are no running jobs. These files get recreated with every new run of the job and are never reused. If the job runs successfully, then only the lookuptable file created during that job run is removed.
Link
command
ls -ltr | grep 'lookuptable*'
ls -ltr | grep 'export*'
rm –rf <files> (If anything returns from above command and any job is not running)

Clearning &PH& files
Cause
There is a &PH& directory in each DataStage project directory. Files in the &PH& directories under DataStage project directories store runtime information when jobs are running and need to be cleared out periodically.
Answer
To clear the &PH& directory from within DataStage:
1.       Ensure there are no DataStage jobs running anywhere on the system by running "ps -ef | grep dsrpc"
2.       From the DataStage Administrator, go to the Projects page, select the project whose file you want to clear and click the Command button. The Command Interface dialog box opens.
3.       Type the following into the command field: CLEAR.FILE &PH& (all uppercase)
4.       Click Execute to run the command and clear the file.


You can also remove the contents of the &PH& directory using the rm command. However, we suggest that this is scheduled for when the system is least used.

** Important **
Please only delete the contents of &PH& directory and not the &PH& directory itself.

You will find a directory for the &PH& in each project that you have created on that server.

You must not remove any contents or directories for the following:
DS_TEMP*
RT_BP*.O
RT_BP*
RT_LOG*
RT_STATUS*
RT_CONFIG*

These directories/files are related to the jobs within your projects that you are running on the server and if you manually remove or edit any of these files you are likely to corrupt the jobs and possibly the projects. These directories exist for each job that has been created within your project. You could ask your developers to review if there are any jobs that are no longer used and can be removed using the DataStage clients. You can additionally clear down some of the RT_LOG* files by clearing down any
large log files that exist for some of the jobs. This can be done from within DataStage Director client:
1.       Select the job
2.       Click on Job > Clear Log > Immediate purge (Clear all Entries).
You can create a shell script to manually delete the files. To ensure there are no locks only delete files that are from finished jobs. You need to make sure the files are older then the longest running job. Generally you can just delete files older then a week.
                DSPROJDIR=/opt/IBM/Ascential/DataStage/Projects
for project in `ls -l ${DSPROJDIR} | grep "^d" | grep -v "lost+found" | awk '{print $9}'`
do
find "/opt/IBM/Ascential/DataStage/Projects/$project/&PH&"  -mtime +13 -exec rm -f {} \;
done
                Link
                                http://www-304.ibm.com/support/docview.wss?uid=swg21457983

Clearning DataSet files
Data sets can be managed using the Data Set Management tool, invoked from the Tools > Data Set Management menu option within DataStage Designer (DataStage Manager in the 7.5 releases.) Alternatively, the 'orchadmin' command line program can be used to perform the same tasks.
The files which store the actual data persist in the locations identified as resource disks in the configuration files. These files are named according to the pattern below:

descriptor.user.host.ssss.pppp.nnnn.pid.time.index.random

descriptor: Name of the data set descriptor file.
user: Your user name.
host: Hostname from which you invoked the job which created the data set.
ssss: 4-digit segment identifier (0000-9999)
pppp: 4-digit partition identifier (0000-9999)
nnnn: 4-digit file identifier (0000-9999) within the partition
pid: Process ID of the job on the host from which you invoked the jop that creates the data set.
time: 8-digit hexadecimal time stamp in seconds.
index: 4-digit number incremented for each file.
random: 8 hexadecimal digits containing a random number to insure unique file names.


For example, suppose that your configuration file contains the following node definitions:

{
    node node0
    {
         fastname "host1"
         pools ""
         resource disk "/orch/s0" {pools ""}
         resource scratchdisk "/scratch" {pools ""}
    }
    node node1
    {
         fastname "host1"
         pools ""
         resource disk "/orch/s0" {pools ""}
         resource scratchdisk "/scratch" {pools ""}
    }
}

A data set named mydata.ds created by a job using this configuration file will contain data in two partitions, one for each processing node declared in the configuration file. Because each processing node contains only a single disk specification, each partition of data would be stored in a single file on each processing node. Following the naming convention shown above, the data file for partition 0 would be located on the host1 machine, in the /orch/s0 filesystem, and the file would be named:

/orch/s0/mydata.ds.user1.host1.0000.0000.0000.1fa98.b61345a4.0000.88dc5aef

The data file for partition 1 data would be similarly named:

/orch/s0/mydata.ds.user1.host1.0000.0001.0000.1fa98.b61345a4.0001.8b3cb144

It is important to understand that the file referenced in the job, called mydata.ds in our example, does not contain any actual data. It is a data set descriptor file, and it contains information about how the data set is constructed. In order for DataStage jobs to access the data, both the descriptor and the actual segment files must exist.


Cleaning up Data Sets

A good plan for managing data sets is to identify the Data Sets that are no longer required, and to use the Data Set Management tool to delete them. If you have the jobs that reference the data sets, you can open each of the data set descriptor files using the Data Set Management tool and then view and delete the data set. If you do not have the jobs, another possible method is to look in the resource disk locations for segment files with very old modification dates. Once you have identified the segment files, you can determine what the data set descriptor file name was.

/orch/s0/mydata.ds.user1.host1.0000.0000.0000.1fa98.b61345a4.0000.88dc5aef

In this example segment file shown above, the highlighted "mydata.ds" is the file name of the data set descriptor. You can then locate this file in your computer with the find command.

find /my_projects/datasets/ -name "mydata.ds" -print

Once you have located the descriptor file, you can then use the Data Set Management tool to view and delete the data set. If someone has already deleted the descriptor file, then the segments have been orphaned. There is no utility or function to recreate the descriptor file. In this situation, you can safely delete all the segment files named with the "mydata.ds" in the file name.


Cleaning up Data Sets from the command line

It is also possible to use the orchadmin executable program to delete data sets. This program is located in $APT_ORCHHOME/bin.

APT_CONFIG_FILE=/opt/IBM/Ascential/DataStage/Configurations/config-2x2.apt
APT_ORCHHOME=/opt/IBM/Ascential/DataStage/PXEngine/bin
cd `cat /.dshome`
. ./dsenv
LD_LIBRARY_PATH=$APT_ORCHHOME/lib:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH
APT_CONFIG_FILE=< /opt/IBM/Ascential/DataStage/Configurations/config-2x2.apt>; export APT_CONFIG_FILE
APT_ORCHHOME=$DSHOME/../PXEngine; export APT_ORCHHOME
PATH=$APT_ORCHHOME/bin:$PATH; export PATH
orchadmin describe -c -p -f -s -v -e /dsd/od251dev/data/incoming/9997/20120414/9997_MRKT_FCTR_VOL.ds
orchadmin delete <full path to descriptor file><datasetname.ds>
Link
http://www.isecor.com/kb/documentation/orchadmin.html

Clear out RT_LOG
                The RT_LOG* files are cleared when you clear the log of a job from the Director

Tuesday, June 5, 2012

Methods of Releasing LOCKS in DataStage

3 methods for releasing locks in Datastage.

     1.via  Director client using cleanup resources.

     2.via UNIX box using DS commands

     3.Via DataStage Admin clients using DS commands

If a job is hanging or failing and not releasing locks, the first thing you  should do is Cleanup resources in Director.

If you  cannot remove the locks in Director's Cleanup resource then you can do it via uv command line. This can be done either via Administrator GUI command line or uvsh on Unix, they both do the same thing, it is just a different interface but there is no difference in the functionality.

Also, when it is a lock related to opening a job in Designer and then Designer crashing, the way to release it is from command line not in Director's cleanup resources.

Cleanup Resources is just for lock associated with running jobs.

---------------------------------------------------------------------------------------------------------
The job is not in running status? If it isn't then you would just need to manually remove the locks.

Please follow this techtip and let us know your results.
http://www-304.ibm.com/support/docview.wss?uid=swg21390366
From Unix/Linux server:

    Log into the server using the dsadm user
    cd to the DSEngine directory
    Enter . ./dsenv to source the dsenv file
    Enter ./bin/uvsh to get into DataStage prompt
    At ">" DataStage engine prompt, enter LOGTO <project name>
    Run LIST.READU EVERY to list all the locks
    Check active record locks under "Item Id" column for job name or RT_CONFIG# or RT_LOG# (# matches the job description number)
    Write down the Inode numbers and user numbers associated with these locks
    Enter LOGTO UV
    Enter UNLOCK INODE inode# USER user# ALL
    You can use Q to get out of DataStage engine