Oracle 12c RMAN backup would create . (hidden files) if you are using NAS storage for the backups. These . files would be converted to regular files once rman stops writing to the backup piece. This is because DNFS is enabled by default in 12c and this is cauing this behavior. Solution is to disable dnfs and you will start seeing normal files even when rman is writing to the backup pieces.
Friday, December 16, 2016
Wednesday, February 19, 2014
Issue with AFPASSWD in 12.1.2
Dont use AFPASSWD to reset oracle user password or apps password. There is a bug (Doc ID 1492939.1) that causes adpatch to fail. To undo this I had to this in one of our environments. There is a patch that fixes this issue but this is a faster and simple workaround.
FNDCPASS apps/xxxxxxxx 0 Y system/xxxxxxxx ALLORACLE xxxxxxxx
FNDCPASS apps/xxxxxxxx 0 Y system/xxxxxxxx ALLORACLE xxxxxxxx
Wednesday, October 23, 2013
Our experience with Apex Listener 2.2 implementation
I try to list out the issues and learnings from our Apex Listener implementation here:
1. Which J2EE server to use to deploy apex listener
A. In my opinion, 10.1.3.X of Oracle Application server or Weblogic 12.1.2 is recommend. GlassFish server was not as straightforward and I was concerned about the support may not be as good as Oracle AS and Weblogic when it comes to raising SRs and troubleshooting. I could be wrong but that was my understanding after going through some readings.
2. What is the equivalent of dads.conf
A. defaults.xml has most of the parameters that dads.conf had in a modplsql implementation. dads.conf is more intuitive compared to defaults.xml.
3. Where are images stored ?
A. I had some confusion over this since apex listener 1.1 had actually stored the images in the war file and there were some documents out there which explained on how to add custom images later without generating the war file. But this is no longer true in Apex Listener 2.2. You store only the location where the images are stored in the war file and you can add any images to that location as long as that location is not changed.
4. During installation weblogic some screens just hang
A. Using vnc server solved this problems. If you used xterm then some screens where you are expected to enter passwords are greyed out or the installation hangs.
5. Noticed that the admin server comes up pretty quickly in 12.1.2 as compared to the 11g Fusion Middleware without using any of the workarounds that were suggested for 11g FMW slow startups of admin servers (you can google on this issue and you will find what those workarounds were)
6. We had a requirement , wherein, the application was using a different context-root (in apex listener or java terminology). This implementation was straightforward in modplsql implementation. If developers had an application that wanted to use http://www.example.com/pls/apex and http://www.example.com/pls/apex2 but both of them pointing to the same database, we had defined an additional dad in dads.conf file for apex2. But in apex listener, we had to deploy apex.war as apex2.war so that apex2.war would be deployed with the context-root apex2.war. There were methods on suggestions on how one could change the context-root of apex.war (using the --context-root option when deploying the war file) but that was only if you had one application that wanted to use a different context-root. But in our case we had multiple applications and both were actually going to the same database but were using different context-roots. I think in all cases, multiple war files are not required (you could have an apache redirect in most cases) but in our case it was required as the application had verified what was the source ip (so one set of applications just allowed you to pass through and the other applications were verifying the source ip). The point I am trying to make is y ou may have to deploy multiple war files to change context-root in some cases and this is a pain point in my opinion when you do upgrades of apex listener (in modplsql implementation, you did not have to bother about this once you had the entries in the dads.conf file).
7. In the load tests that were done, apex listener was better than modplsql implementation in terms of handling the load. At the least ,it is as good as modplsql implementation. I did not have to modify the defaults.xml and load test was run for 100 concurrent users successfully.
8. Configuring for multiple databases using the map-url and workspace-id features of apex.war exists but I have not been to get it to work successfully. I will have to revisit this but again this is one of the features that is not straightforward in an apex listener implementation (compared to modplsql implementation - just another entry in dads.conf file would have been enough)
9. You can integrate Apex applications with Oracle Ebusiness Applications and apex listener is the only supported way of doing it. This is one more reason why you would want to move to apex listener. But you can still integrate modplsql based apex applications with oracle ebusiness ( it works but not supported and recommended)
10. The support at forums.oracle.com for Apex Listener is very good and you usually get good and valid responses from here if there are any implementation questions.
1. Which J2EE server to use to deploy apex listener
A. In my opinion, 10.1.3.X of Oracle Application server or Weblogic 12.1.2 is recommend. GlassFish server was not as straightforward and I was concerned about the support may not be as good as Oracle AS and Weblogic when it comes to raising SRs and troubleshooting. I could be wrong but that was my understanding after going through some readings.
2. What is the equivalent of dads.conf
A. defaults.xml has most of the parameters that dads.conf had in a modplsql implementation. dads.conf is more intuitive compared to defaults.xml.
3. Where are images stored ?
A. I had some confusion over this since apex listener 1.1 had actually stored the images in the war file and there were some documents out there which explained on how to add custom images later without generating the war file. But this is no longer true in Apex Listener 2.2. You store only the location where the images are stored in the war file and you can add any images to that location as long as that location is not changed.
4. During installation weblogic some screens just hang
A. Using vnc server solved this problems. If you used xterm then some screens where you are expected to enter passwords are greyed out or the installation hangs.
5. Noticed that the admin server comes up pretty quickly in 12.1.2 as compared to the 11g Fusion Middleware without using any of the workarounds that were suggested for 11g FMW slow startups of admin servers (you can google on this issue and you will find what those workarounds were)
6. We had a requirement , wherein, the application was using a different context-root (in apex listener or java terminology). This implementation was straightforward in modplsql implementation. If developers had an application that wanted to use http://www.example.com/pls/apex and http://www.example.com/pls/apex2 but both of them pointing to the same database, we had defined an additional dad in dads.conf file for apex2. But in apex listener, we had to deploy apex.war as apex2.war so that apex2.war would be deployed with the context-root apex2.war. There were methods on suggestions on how one could change the context-root of apex.war (using the --context-root option when deploying the war file) but that was only if you had one application that wanted to use a different context-root. But in our case we had multiple applications and both were actually going to the same database but were using different context-roots. I think in all cases, multiple war files are not required (you could have an apache redirect in most cases) but in our case it was required as the application had verified what was the source ip (so one set of applications just allowed you to pass through and the other applications were verifying the source ip). The point I am trying to make is y ou may have to deploy multiple war files to change context-root in some cases and this is a pain point in my opinion when you do upgrades of apex listener (in modplsql implementation, you did not have to bother about this once you had the entries in the dads.conf file).
7. In the load tests that were done, apex listener was better than modplsql implementation in terms of handling the load. At the least ,it is as good as modplsql implementation. I did not have to modify the defaults.xml and load test was run for 100 concurrent users successfully.
8. Configuring for multiple databases using the map-url and workspace-id features of apex.war exists but I have not been to get it to work successfully. I will have to revisit this but again this is one of the features that is not straightforward in an apex listener implementation (compared to modplsql implementation - just another entry in dads.conf file would have been enough)
9. You can integrate Apex applications with Oracle Ebusiness Applications and apex listener is the only supported way of doing it. This is one more reason why you would want to move to apex listener. But you can still integrate modplsql based apex applications with oracle ebusiness ( it works but not supported and recommended)
10. The support at forums.oracle.com for Apex Listener is very good and you usually get good and valid responses from here if there are any implementation questions.
Saturday, May 25, 2013
FRM-91111,FRM-10039 when trying to involke frmbld.sh
We were trying to open forms in an ebusiness 12.1.2 environment and we would get this error
frmbld.sh
FRM-91111: Internal Error: window system startup failure.
FRM-10039: Unable to start up the Form Builder.
Solution was to unset NLS_LANG (export NLS_LANG= ) and t.hen we were able to open forms builder
Eventhough some notes suggested that NLS_LANG can be set but it should not UTF8 (ours was American_America.WE8ISO8859P15)
frmbld.sh
FRM-91111: Internal Error: window system startup failure.
FRM-10039: Unable to start up the Form Builder.
Solution was to unset NLS_LANG (export NLS_LANG= ) and t.hen we were able to open forms builder
Eventhough some notes suggested that NLS_LANG can be set but it should not UTF8 (ours was American_America.WE8ISO8859P15)
Friday, May 17, 2013
Interesting issue with mount points, single instance (non-rac)
We had a 2 node cluster installation (11.2.0.2) (ASM + GI standard installation). This supported our 2 node rac (2 rac databases on this boxes).
To this combination we added a 3rd database which was non-rac. The database files were on NAS and just like any of our single database (non-rac) databases, we had the following mount options
rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp 0 0
When we tried to mount the asm diskgroup that supported this database, the asm alert log reported an issue with mount options and the diskgroup would not mount. The exact error was
WARNING:NFS file system /db/of3uat mounted with incorrect options(rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=vfiler9-3248)
WARNING:Expected NFS mount options: rsize>=32768,wsize>=32768,hard,noac/actimeo=0
Searched on metalink and found note 1367321.1. Per this note (though no exact hit) it was ok to get the above warning.
But metalink note 359515.1 had the solution (athough not directly mentioned in the note).
Per this noted, if you are on non-rac, the mount options you need to use for datafiles were
rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp
and for rac it was
rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,actimeo=0, vers=3,timeo=600
So eventhough we are trying to bring up a non-rac database, since we are using the GI + ASM infrastructure that was originally installed for rac databases, we had to mount the file system that supported the non-rac database also using the same options that are used for rac (rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,actimeo=0, vers=3,timeo=600)
When this change was made, we were able to mount the diskgroup without any issues.
To this combination we added a 3rd database which was non-rac. The database files were on NAS and just like any of our single database (non-rac) databases, we had the following mount options
rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp 0 0
When we tried to mount the asm diskgroup that supported this database, the asm alert log reported an issue with mount options and the diskgroup would not mount. The exact error was
WARNING:NFS file system /db/of3uat mounted with incorrect options(rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=vfiler9-3248)
WARNING:Expected NFS mount options: rsize>=32768,wsize>=32768,hard,noac/actimeo=0
Searched on metalink and found note 1367321.1. Per this note (though no exact hit) it was ok to get the above warning.
But metalink note 359515.1 had the solution (athough not directly mentioned in the note).
Per this noted, if you are on non-rac, the mount options you need to use for datafiles were
rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp
and for rac it was
rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,actimeo=0, vers=3,timeo=600
So eventhough we are trying to bring up a non-rac database, since we are using the GI + ASM infrastructure that was originally installed for rac databases, we had to mount the file system that supported the non-rac database also using the same options that are used for rac (rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,actimeo=0, vers=3,timeo=600)
When this change was made, we were able to mount the diskgroup without any issues.
Thursday, March 21, 2013
Command to copy concurent queue definition between ebzi environments.
FNDLOAD apps/appspass 0 Y DOWNLOAD
$FND_TOP/patch/115/import/afcpque.lct feeder_cm.ldt QUEUE
APPLICATION_SHORT_NAME=MY_APP CONCURRENT_QUEUE_NAME="MY QUEUE"
FNDLOAD apps/appspass 0 Y UPLOAD $FND_TOP/patch/115/import/afcpque.lct feeder_cm.ldt QUEUE APPLICATION_SHORT_NAME=MY_APP CONCURRENT_QUEUE_NAME="MY_QUEUE"
the above fndload commands can be used to download and upload concurrent queues between environnments. At our work, we have a frequent requirement where users want to run a bunch of concurrent programs but do not want to block the standard manager or other users. They ask us to create special concurrent managers for this purpose and migrate it across the 10 environments we have. Before discovering the above, we used to do each of the steps manually in each environment. Te above commands make it easier to migrate. This does copy the specalization and workshift rules. However, the exclusions from standard manager and other managers (that should be made when a concurrent program has to exclusively run from a particular manager) are not done. That step is still a manual step.
FNDLOAD apps/appspass 0 Y UPLOAD $FND_TOP/patch/115/import/afcpque.lct feeder_cm.ldt QUEUE APPLICATION_SHORT_NAME=MY_APP CONCURRENT_QUEUE_NAME="MY_QUEUE"
the above fndload commands can be used to download and upload concurrent queues between environnments. At our work, we have a frequent requirement where users want to run a bunch of concurrent programs but do not want to block the standard manager or other users. They ask us to create special concurrent managers for this purpose and migrate it across the 10 environments we have. Before discovering the above, we used to do each of the steps manually in each environment. Te above commands make it easier to migrate. This does copy the specalization and workshift rules. However, the exclusions from standard manager and other managers (that should be made when a concurrent program has to exclusively run from a particular manager) are not done. That step is still a manual step.
Thursday, January 24, 2013
Migrating Oracle E-Business Suite R12 from Linux 32-bit to Linux 64-bit
I try to document some of the issues we faced while migrating from 32 bit to 64 bit for application servers
LINUX_X86-64
1. The first misconception was that if we move to 64 bit from 32 bit, we would not be able to patch the application since there are not 64 bit patches available for all patches. There were conflicting opinions from Oracle Support (depends on the guys who pick your SR) that this is true (you cannot patch 64 bit environments with 32 bit patches and you would have to install 64 bit executables). This is not true and 32 bit patches can be applied on 64 bit platforms (this is confirmed in Steven Chan's blog too). In fact, Oracle probably does not specifically release 64 bit patches for Ebusiness suite.
2. Our Unix sysadmin did not install linux32 and this caused issues with all the shutdown and startup scripts. All of them check for the existence of linux32 before executing the stop and start commands. Exceptions are concurrent manager and apps listener startup/shutdown scripts.
3. Our Unix admin installed linux32 but we still had issues during startup. This was because he probably did not install it properly or got a wrong version. After installing linux32, we did not see issues with startup scripts but the command would just hang and timeout finally.
This was confirmed by running these commands
inux32
usage: linux32 [--3gb] [--4gb] program args ...
Default is --3gb to limit the address space of the 32bit children to 3GB
/u01/app/applmgr
applmgr@ofapdev21:OF1TCH> linux32 --4gb
Memory fault
/u01/app/applmgr
applmgr@ofapdev21:OF1TCH> linux32 --3gb
Memory fault
/u01/app/applmgr
applmgr@ofapdev21:OF1TCH> uname -m
x86_64
On a working system, where linux32 works properly the output should be like this
$ uname -m
x86_64
$ linux32
-sh-3.2$ uname -m
i686
-sh-3.2$
Our unix admin reinstalled linux32 and we stopped getting these errors anymore.
4. The other issue (this is silly and may be not everyone will face it) was rpc.statd process was not started on these application servers and we have a shared appl top (NAS) and even our INST_TOP was on NAS (not shared between apps servers though). This caused some of the startup scripts not to acquire locks on files and these were seen in $LOG_HOME/ora/10.1.3/opmn/*.err files. Related issue was the update snaphot (through adadmin) never completed because it was waiting for acquiring a lock. I would see message like this
Trying to obtain a lock...
Deleting Temp Global Snapshot files ...Start time:Sun Oct 28 2012 18:05:39
Deleting Temp Global Snapshot Bugfixes ...Start time:Sun Oct 28 2012 18:05:39
Updating Global Snapshot files ...Start time:Sun Oct 28 2012 18:05:39
Total No. of records to be processed = 758374
5. The other thing that I noticed is that the startup/shutdown take considerably longer time on 64 bit boxes compared to 32 bit boxes (both have same memory). Still need to investigate this.
6. In the document " Migrating Oracle E-Business Suite R12 from Linux 32-bit to Linux 64-bit [Doc Id 471566.1]" one of the steps is Update CTXORIG.xml.
Instead of this why cant I directly update the context file to have this value
and run autoconfig.
What is the difference between changing this in CONTEXT FILE vs ctxorig.xml. In fact, ctxorig.xml is generated by autoconfig and it would be overwritten even if I changed the ctxorig.xml and do not change context file and then run autoconfig
That was the question I had and for that Oracle had this response.
I found you might find issues such as the one described in Note After Running Afcfgclone.Pl The Library Files On Apps Tier Disappear (Doc ID 564721.1) if you fail to set the right platform when performing the migration in the CTXORIG.xml. Per SystemCheck.java the check for the right platform is made first in CTXORIG.xml file.
Subscribe to:
Posts (Atom)