How to make BDB to auto remove log file


Jephe Wu - http://linuxtechres.blogspot.com

Objective: to understand how BDB (DB4) log file remove works and automatically remove old logfiles to prevent disk space full
background:  Ubuntu 8.04.2 , Berkeley DB 4.2.52, OpenLDAP


Steps:

1.  BDB version used by openldap

root@ldap1:~/ # ldd /usr/sbin/slapd
linux-gate.so.1 =>  (0xb7faa000)
libldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2 (0xb7f62000)
liblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2 (0xb7f55000)
libdb-4.2.so => /usr/lib/libdb-4.2.so (0xb7e7c000)


root@ldap1:~/ # dpkg -l| grep db
ii  console-tools                              1:0.2.3dbs-65ubuntu7        Linux console and font utilities
ii  dbus                                       1.1.20-1ubuntu3.2           simple interprocess messaging system
ii  libconsole                                 1:0.2.3dbs-65ubuntu7        Shared libraries for Linux console and font
ii  libdb4.2                                   4.2.52+dfsg-4               Berkeley v4.2 Database Libraries [runtime]


2. log file removal
http://sepp.oetiker.ch/db-4.2.52-mo/ref/transapp/logfile.html

Log files may be removed at any time, as long as:


  • the log file is not involved in an active transaction.
  • a checkpoint has been written subsequent to the log file's creation.
  • the log file is not the only log file in the environment.


method 3:

Call the DB_ENV->set_flags method from the application, with the DB_LOG_AUTOREMOVE (http://sepp.oetiker.ch/db-4.2.52-mo/api_c/env_set_flags.html#DB_LOG_AUTOREMOVE) flag, to remove any log files that are no longer needed on an ongoing basis. With this configuration, Berkeley DB will automatically remove log files, and the application will not have an opportunity to copy the log files to backup media.

Two ways to implement checkpoint (besides DB_LOG_AUTOREMOVE flag in DB_CONFIG)


a. checkpoint settings in slapd.conf

checkpoint kbyte min
The checkpoint directive defines the time between checkpoint operations in BDB (the database can only be recovered from the last checkpoint).

The frequency of checkpointing determines the time during which data may be unrecoverable by BDB in the event of a system failure. If NOT using the dbnosync this time could be set to a reasonably long period, say, 10 mins or more, if the dbnosync directive is being used 5 - 15 mins or less if practical. kbytes is the number of kilobytes written to the directory and min is the time in minutes. Whichever occurs first determines the period between checkpoints.

OpenLDAP default is NO CHECKPOINTING - you should always supply a checkpoint directive. See the BDB Chapter 12 Section 15 documentation for more information. This directive may be replaced by using DB_CONFIG file with the txn_checkpoint directive. Examples:

checkpoint 128 15
# check point whenever 128k data bytes written or
# 15 minutes has elapsed whichever occurs first


b.  txn_checkpoint in DB_CONFIG file


# more DB_CONFIG
# see more detail config parameter at http://sepp.oetiker.ch/db-4.2.52-mo/ref/env/intro.html

set_flags DB_LOG_AUTOREMOVE
txn_checkpoint 1024 5 0



DB_CONFIG example:
http://www.zytrax.com/books/ldap/ch6/bdb.html#db-config




root@ldap1:/etc/ldap/ # grep checkpoint slapd.conf
# first DIT definition
database bdb
...
# DIT will act as a provider
overlay syncprov
checkpoint  1024 5
syncprov-checkpoint 100 10


3. working configuration example:
The following configuration makes logfile auto remove possible in my testing environment.


  • DB_CONFIG (sitting at the same directory as BDB database files)

set_cachesize 0 2097152 0
# default cache size is 256KB
# sets a database cache of 0G(first column) + 2M(2*1024*1024=2097152) and 
# do not allows fragmentation 
# does NOT replace slapd.conf cachesize 
# this is a database parameter, for slapd.conf cachesize paramter, refer to
# http://www.zytrax.com/books/ldap/ch6/bdb.html#cachesize
# see http://sepp.oetiker.ch/db-4.2.52-mo/api_c/env_set_cachesize.html
# The database environment's cache size may also be set using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_cachesize", one or more whitespace characters, and the cache size specified in three parts: the gigabytes of cache, the additional bytes of cache, and the number of caches, also separated by whitespace characters. For example, "set_cachesize 2 524288000 3" would create a 2.5GB logical cache, split between three physical caches. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time.
set_lk_max_objects 1500 
set_lk_max_locks 1500
set_lk_max_lockers 1500
set_flags DB_LOG_AUTOREMOVE
txn_checkpoint 1024 5 0
# replaces checkpoint in slap.conf
# writes checkpoint if 128K written or every 15 mins
# 0 = no writes - no update 

  • slapd.conf  (put the following checkpoint in the every BDB definition)
checkpoint  1024 5

4. References:


http://www.openldap.org/doc/admin24/maintenance.html

Administrators can change the size limit of a single log file (by default 10MB), and have old log files removed automatically, by setting up DB environment (see below). The reason Berkeley DB never deletes any log files by default is that the administrator may wish to backup the log files before removal to make database recovery possible even after a catastrophic failure, such as file system corruption.

Log file names are log.XXXXXXXXXX (X is a digit). By default the log files are located in the BDB backend directory. The db_archive tool knows what log files are used in current transactions, and what are not. Administrators can move unused log files to a backup media, and delete them. To have them removed automatically, place set_flags DB_LOG_AUTOREMOVE directive in DB_CONFIG.

Note: If the log files are removed automatically, recovery after a catastrophic failure is likely to be impossible.


http://www.zytrax.com/books/ldap/ch6/bdb.html#samples

cachesize  in slapd.conf
Format:
cachesize integer
The cachesize directive defines the number of entries that the LDAP backend will maintain in memory. Do not confuse this directive with the BDB set_cachesize directive - they control different behaviours.

For maximum performance this figure should be as high as practical or as close as practical to the number of records maintained in the directory. The default is 1000. Examples:

cachesize 10000
# LDAP maintains 10,000 entries in memory 
See also Performance chapter.


http://www.openldap.org/faq/data/cache/1072.html


How to rebuild Oracle database controlfile

Jephe Wu - http://linuxtechres.blogspot.com

Objective:  rebuild database controlfile without any backup
Environment: Oracle database enterprise edition 11.2.0.3 64bit, lost controlfile without any RMAN backup, datafile and online redo log are okay. need to rebuild control file manually.


Steps:

Preparation:

  • Find out all files path such as redo log, datafile
  • Find out database SID from /etc/oratab
  • calculate redo log file size


1. prepare rebuild script.

-- Below are two sets of SQL statements, each of which creates a new
-- control file and uses it to open the database. The first set opens
-- the database with the NORESETLOGS option and should be used only if
-- the current versions of all online logs are available. The second
-- set opens the database with the RESETLOGS option and should be used
-- if online logs are unavailable.
-- The appropriate set of statements can be copied from the trace into
-- a script file, edited as necessary, and executed when there is a
-- need to re-create the control file.
--
--     Set #1. NORESETLOGS case
--
-- The following commands will create a new control file and use it
-- to open the database.
-- Data used by Recovery Manager will be lost.
-- Additional logs may be required for media recovery of offline
-- Use this only if the current versions of all online logs are
-- available.

--     Set #2. RESETLOGS case
--
-- The following commands will create a new control file and use it
-- to open the database.
-- Data used by Recovery Manager will be lost.
-- The contents of online logs will be lost and all backups will
-- be invalidated. Use this only if online logs are damaged.


$ more controlfile_rebuild.sql

STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "sm" NORESETLOGS FORCE LOGGING ARCHIVELOG
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 1200
    MAXINSTANCES 8
    MAXLOGHISTORY 584
LOGFILE
  GROUP 1 (
    '/u01/oradata/sm/redo01.log'
  ) SIZE 50M BLOCKSIZE 512,
  GROUP 2 (
    '/u01/oradata/sm/redo02.log'
  ) SIZE 50M BLOCKSIZE 512,
  GROUP 3 (
    '/u01/oradata/sm/redo03.log'
  ) SIZE 50M BLOCKSIZE 512
DATAFILE
  '/u02/oradata/sm/system01.dbf',
  '/u02/oradata/sm/sysaux01.dbf',
  '/u02/oradata/sm/undotbs01.dbf',
  '/u02/oradata/sm/users01.dbf'
CHARACTER SET AL32UTF8
;

RECOVER DATABASE

ALTER SYSTEM ARCHIVE LOG ALL;

ALTER DATABASE OPEN;


[oracle@oratest sm]$ sqlplus / as sysdba @controlrebuild.sql 

SQL*Plus: Release 11.2.0.3.0 Production on Mon Dec 3 15:50:39 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size    2235208 bytes
Variable Size 2315257016 bytes
Database Buffers 1946157056 bytes
Redo Buffers   12132352 bytes

Control file created.

ORA-00283: recovery session canceled due to errors
ORA-00264: no recovery required



System altered.


Database altered.

SQL> select status from v$instance;

STATUS
------------
OPEN

SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/u02/oradata/sm/system01.dbf
/u02/oradata/sm/sysaux01.dbf
/u02/oradata/sm/undotbs01.dbf
/u02/oradata/sm/users01.dbf


SQL>  show parameter control

NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time     integer 7
control_files     string /u01/oradata/sm/control01.ctl,
 /u01/oradata/sm/control02.ctl
control_management_pack_access     string DIAGNOSTIC+TUNING


SQL> ALTER TABLESPACE TEMP ADD TEMPFILE '/u02/oradata/sm/temp02.dbf' size 10m autoextend on next 1m;

Tablespace altered.

SQL> select name from v$tempfile;

NAME
--------------------------------------------------------------------------------
/u02/oradata/sm/temp02.dbf

sql> !rm -f /u02/oradata/sm/temp01.dbf